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Preface 


The  purpose  of  this  study  was  to  determine  the  extent 
to  which  Air  Force  Medical  Systems  Office  staffing  source 
was  related  to  the  level  of  quality  of  services  provided  in 
six  medical  centers.  In  these  days  of  increased  emphasis  on 
quality  management,  we  found  a  deep  concern  for  providing 
the  best  medical  information  system  customer  service  at 
every  leadership  level  in  the  Air  Force  Medical  Service. 

Several  people  provided  us  with  considerable  assistance 
and  guidance  in  this  research  project  and  deserve  to  be 
mentioned.  Captain  Herb  Smaltz  at  USAF  Medical  Center 
Hright-Patterson  spent  several  sessions  brainstorming  with 
us  to  find  a  manageable  and  interesting  thesis  topic.  Major 
Pat  Lewis  smoothed  the  way  to  our  receiving  research  spon¬ 
sorship  from  AFMSA  and  provided  invaluable  guidance,  back¬ 
ground,  and  suggestions  for  further  sources  of  information. 
He  are  also  grateful  to  the  CIOs  at  all  six  USAF  medical 
centers  for  their  interest,  cooperation,  and  time  in  provid¬ 
ing  us  with  factual  information  and  in  the  complicated 
process  of  distributing  surveys  to  their  users.  He  would 
also  like  to  thank  Colonel  Obuchowski  and  his  staff  for 
their  sponsorship  and  assistance  and  especially  Nicki  for 
her  patience  with  our  dozens  of  phone  calls  and  diligence  in 
obtaining  her  boss's  blessing  on  this  project.  Last  but 
certainly  not  least,  we  are  greatly  indebted  to  Lt  Col  Clyde 
Caufield  and  Dr.  Guy  Shane,  our  thesis  advisors,  for  their 
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persistent  guidance  and  encouragement.  As  a  result  of  their 
talents  as  educators,  we  believe  we  are  able  to  conduct  good 
research. 

Shelley  D.  Christian  and  William  K.  Dorr 

Throughout  my  time  here  at  AFIT,  and  especially  while 
putting  in  the  grueling  hours  for  this  thesis,  I  have  been 
constantly  reminded  of  the  foremost  importance  of  my  family. 
My  husband,  Jim,  was  unwavering  in  his  support  for  my  career 
aspirations  and  understanding  of  the  hardships  involved  in 
our  2000  mile  separation  because  of  my  tour  at  AFIT.  Our 
oldest  daughter.  Shannon,  faithfully  helped  her  dad  in 
maintaining  our  California  household.  I  hope  someday 
Carrie,  our  almost  4  year  old,  will  understand  why  she 
couldn't  see  her  daddy  and  sister  very  often  and  why  her 
mommy  had  to  study  so  much.  Their  loyalty  and  love  helped 
me  to  remember  that  in  the  grand  scheme  of  things  family  is 
what  matters  most.  I  lovingly  dedicate  this  project  to 
them. 

Shelley  D.  Christian 

I  wish  to  thank  my  wife,  Deirdre,  for  her  never  ending 
support  as  I  struggled  to  complete  my  thesis  here  at  AFIT. 

As  always  she  kept  the  family  unit  intact  while  I  was  thor¬ 
oughly  immersed  in  the  AFIT  experience.  A  special  thank  you 
is  extended  to  my  children  Marian  and  Bill  who  spent  count¬ 
less  hours  assembling  research  questionnaires  and  stuffing 
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envelopes  with  minimal  supervision.  Their  commitment  and 
understanding  helped  me  to  stay  focused  and  see  this  project 
through  to  completion. 

William  K.  Dorr 
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Abstract 

This  study  measured  the  relationship  between  Medical 
Systems  Office  staffing  source  (i.e.,  military  medical, 
military  communications/ computer ,  civilian  government,  and 
contractor  personnel)  and  quality  of  automated  information 
services  provided  at  CONUS  USAF  medical  centers.  A  litera¬ 
ture  search  determined  that  information  system  quality  is 
best  measured  on  eight  dimensions:  performance,  features, 
reliability,  conformance,  durability,  serviceability,  aes¬ 
thetics,  and  perceived  quality.  The  CIO  at  each  medical 
center  was  surveyed  to  determine  the  level  of  system  con¬ 
formance  to  Air  Force  standards  with  a  100  percent  response 
rate  and  to  gather  data  on  each  MSO  staff.  Users  at  the 
medical  centers  were  also  surveyed  in  order  to  measure 
performance,  features,  reliability,  serviceability,  aesthet¬ 
ics,  and  perceived  quality.  Durability  was  not  measured  in 
this  study.  The  users  were  given  the  opportunity  to  evalu¬ 
ate  up  to  three  different  information  systems  in  operation 
at  their  medical  center  in  several  areas.  Approximately 
42.5  percent  of  the  user  survey  population  responded.  No 
relationship  was  evident  between  MSO  staffing  source  or 
source  of  staff  supervision  and  the  level  of  quality  of 
information  services.  Further,  the  level  of  quality  was 
unrelated  to  the  types  of  information  systems  in  operation. 
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STAFFING  SOURCES  OF  USAF  MEDICAL  CENTER 
SYSTEMS  OFFICES:  A  STUDY  OF  THEIR  RELATION  TO 
INFORMATION  SYSTEMS  QUALITY 


I.  Introduction 


General  Issue 

In  1990,  Captain  Alan  R.  Constantian  conducted  a  feasi¬ 
bility  assessment  of  executive  information  systems  for  USAF 
hospital  administrators.  After  analyzing  data  collected 
from  Air  Force  hospitals  within  the  continental  United 
States  (CONUS),  he  recommended  "the  training  and  educational 
opportunities  for  medical  systems  officers  and  their  staffs 
must  be  seriously  evaluated"  (Constantian, 1990 : 155 ) .  His 
data  revealed  that  information  system  staffs  had  a  low  level 
of  confidence  in  their  own  abilities.  Further,  he  found 
that  systems  end  users  (see  Definitions,  below)  generally 
did  not  feel  the  information  system  staff  could  adequately 
provide  support  for  technological  advances,  such  as  the 
addition  of  an  executive  information  system 
(Constantian, 1990: 155) . 

Armed  with  Constantian’ s  findings  and  recommendations, 
the  researchers  approached  Captain  Detlev  H.  Smaltz,  chief 
information  officer  at  Wright-Patterson  Air  Force  Base 
Medical  Center.  The  goal  in  this  initial  meeting  was  to  get 
Capt  Smaltz's  opinions  on  the  topic  of  information  system 
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(IS)  staff  training  opportunities  and  skill  levels.  The 
interview  revealed  that  this  topic  was  the  subject  of  much 
debate  in  the  Air  Force  Medical  Service  (Smaltz, 1991) . 

Smaltz  stated  that  U.S.  Air  Force  Program  Action  Direc¬ 
tive  (PAD)  90-4,  which  has  been  approved  for  implementation, 
would  have  profound  effects  on  the  staffing  of  USAF  hospital 
information  systems  (HIS)  (Smaltz ,1991) .  The  PAD  has  been 
approved  by  the  Air  Force  Office  of  Medical  Systems  and 
implements  a  new  plan  for  management  of  computers  throughout 
the  Air  Force.  When  fully  operational,  all  USAF  computer 
systems  (medical  and  nonmedical)  will  be  staffed  by  person¬ 
nel  with  communications-computer  specialty  codes.  The 
computer  people  will  remain  under  the  supervision  of  the 
communications  community,  but  will  physically  work  wherever 
the  actual  computer  hardware  is  located  (Department  of  the 
Air  Force, 1990a) . 

According  to  Smaltz,  the  greatest  impact  of  the  PAD  at 
Wright-Patterson  AFB  Medical  Center  will  be  to  change  organ¬ 
izational  responsibility.  The  Medical  Systems  Office  (MSO) 
is  presently  staffed  with  communications-computer  people  who 
report  directly  to  the  medical  administrative  portion  of  the 
hospital.  Other  USAF  Medical  Centers  are  staffed  using 
alternative  methods  and  will  be  affected  differently  by  the 
PAD  ( Smaltz, 1991 ) . 

Specifically,  Capt  Smaltz  informed  the  researchers  that 
at  least  one  medical  center  had  contracted  out  the 
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management  of  its  information  system  with  a  civilian  corpo¬ 
ration.  Further,  other  medical  centers  were  staffed  with 
medical  administrative  specialists  who  had  received  addi¬ 
tional  computer  training  (Smaltz ,1991) . 

With  evidence  of  a  wide  range  of  information  systems 
staffing  sources,  the  researchers  see  a  potential  for  a  wide 
range  of  quality  levels  within  the  information  systems 
themselves.  Therefore,  the  issue  in  this  research  is  the 
extent  to  which  the  quality  of  information  system  support  is 
related  to  the  staffing  source  in  Air  Force  medical  center 
systems  offices. 

Research  Objective 

Under  the  sponsorship  of  the  Air  Force  Medical  Systems 
Agency  (AFMSA) ,  formerly  the  Air  Force  Office  of  Medical 
Support  (AFOMS),  the  researchers  examined  the  various  infor¬ 
mation  systems  staffing  methods  and  quantified  the  level  of 
quality  in  the  information  systems  at  each  of  the  six  USAF 
medical  centers  in  the  continental  United  States  (CONUS) . 

The  results  of  this  study  will  reveal  the  extent  of  the 
relationship  between  staffing  source  and  quality  of  the 
information  system. 

Specific  Research  Question 

To  what  extent  is  there  a  relationship  between  the 
source  of  MSO  staffing  and  the  level  of  quality  of  the 
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services  provided  by  medical  systems  offices  in  United 
States  Air  Force  medical  centers? 

Investigative  Questions 

In  order  to  answer  the  above  specific  research  ques¬ 
tion,  the  following  investigative  questions  need  to  be 
answered. 

Question  1.  What  is  the  source  of  each  Medical  Systems 
Office  staff  member  at  each  medical  center  (i.e.,  medical 
personnel  with  additional  computer  training,  communications 
squadron  personnel,  or  contractor)? 

Question  2.  What  are  the  quality  indicators  necessary 
to  measure  the  quality  of  information  systems  at  USAF  Medi¬ 
cal  Centers? 

Question  3.  What  are  the  performance  standards  and 
goals  of  the  USAF  Medical  Service  as  set  by  the  Air  Force 
Medical  Systems  Agency  concerning  information  systems  at 
USAF  Medical  Centers? 

Question  4.  To  what  extent  are  the  Medical  Systems 
Offices  meeting  the  standards  and  goals  set  by  the  Air  Force 
for  the  operation  of  medical  information  systems? 

Question  5.  To  what  extent  do  the  customers  feel  that 
the  Medical  Systems  Office  is  providing  them  with  quality 
technical  support  in  information  systems? 
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Definitions 

In  order  to  avoid  any  misunderstandings  of  terminology, 
some  key  terms  will  be  defined  here  as  a  means  to  eliminate 
possible  obstacles  to  the  reader's  clear  assessment  of  the 
research. 

Information  System  (IS).  An  information  system  is  an 
entity  composed  of  hardware,  software,  data,  procedures,  and 
people.  The  information  system's  functions  are  to  collect, 
transmit,  process,  and  store  data,  and  retrieve  and 
distribute  information  to  the  system's  users  (Ahituv  and 
Neumann, 1990: 2) . 

Chief  Information  Officer  (CIO).  The  Chief  Information 
Officer  is  the  director  and  top  manager  of  an  organization's 
information  system  (Ahituv  and  Neumann, 1990: 200 ) . 

End  User.  End  users  are  those  individuals  who  directly 
use  terminals  or  microcomputers  to  access  data  and  programs. 
They  are  the  information  system's  customers  (Ahituv  and 
Neumann, 1990 : 547 ) . 

Outsourcing .  Outsourcing  is  the  practice  of  hiring  a 
group  from  outside  the  organization  to  manage  and  oversee 
the  operations  of  the  organization's  information  system. 

The  contractors  do  not  fall  under  the  administrative 
auspices  of  the  organization  (Hard,  1991;  Kelly, 1990; 

Wil liamson, 1991) . 

Hospital  Information  System  (HIS).  A  hospital  informa¬ 


tion  system  is  a  group  of  computers  and/or  software  designed 
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to  support  the  flow  of  interdepartmental  information  (both 
administrative  and  clinical)  within  hospitals  (Abrami  and 
Sneider , 1985 : 44) . 

Medical  System  Office  (MSP) .  A  medical  system  office 
refers  to  a  hospital's  information  system  staff  and  the 
physical  location  where  they  perform  their  duties. 

USAF  Medical  Center.  Air  Force  Medical  Centers  have  the 
most  inpatient  beds  (the  gauge  by  which  the  size  of  a  hospi¬ 
tal  is  measured)  of  the  three  classes  of  Air  Force  hospi¬ 
tals.  They  provide  a  wider  range  of  medical  services  than 
the  smaller  hospitals,  receive  referrals  from  the  lower 
level  hospitals,  and  provide  specialty  training  to  medical 
professionals.  The  Air  Force  has  six  medical  centers  in  the 
CONUS:  Wilford  Hall  Medical  Center  (Lackland  AFB ,  Texas), 

Wright-Patterson  Medical  Center  (Wright-Patterson  AFB, 

Ohio),  David  Grant  Medical  Center  (Travis  AFB,  California), 
Malcolm  Grow  Medical  Center  (Andrews  AFB,  Maryland),  Keesler 
Medical  Center  (Keesler  AFB,  Mississippi),  and  Scott  Medical 
Center  (Scott  AFB,  Illinois)  (AFR  168-4 , 1990 : 2-9a) . 

Air  Force  Medical  Systems  Agency  ( AFMSA ) .  The  USAF 
Medical  Systems  Agency,  formerly  the  Air  Force  Office  of 
Medical  Support  or  AFOMS,  located  at  Brooks  AFB,  Texas,  and 
specifically  the  Medical  Service  Information  Systems  Divi¬ 
sion,  functions  to  ensure  standardization  and  equality  among 
Air  Force  medical  treatment  facility  information  systems 
(AFR  168-4,1990:15). 


6 


Scope  and  Limitations 


The  scope  of  this  research  is  limited  by  the  population 
under  study,  i.e.,  the  six  medical  centers.  Although  there 
are  40  Air  Force  hospitals  of  varying  size  in  the  CONUS,  the 
researchers  deemed  it  appropriate  to  limit  this  study  to 
medical  centers  for  two  reasons.  First,  the  medical  centers 
have  more  highly  developed  information  systems  with  large 
full-time  staffs  due  to  the  nature  of  their  medical  mission. 
Second,  as  will  be  discussed  in  the  third  chapter,  personal 
interviews  will  be  administered  to  CIOs  in  order  to  obtain 
pertinent  data  concerning  IS  services  and  IS  staff  members 
at  the  medical  facilities.  A  limited  population  of  the  six 
medical  centers  helped  to  ensure  that  this  project  dealt 
with  sites  with  similar  missions,  range  of  services,  and 
available  resources.  A  similar  research  project  using  a 
larger  number  of  medical  facilities  in  the  population  could 
be  a  topic  for  further  research. 

Summary 

This  chapter  identified  an  issue  of  concern  to  the  USAF 
Medical  Service:  the  relationship  between  information  sys¬ 
tems  staffing  source  and  quality  of  the  information  system. 
Furthermore,  specific  investigative  questions  were  identi¬ 
fied  pertaining  to  the  research  question  and  key  definitions 
were  provided.  The  remainder  of  this  thesis  will  be  dedi¬ 
cated  to  responding  to  those  investigative  questions  that 
could  be  answered  from  secondary  data  sources,  to  describing 
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the  research  methodology  that  will  be  used  to  answer  those 
questions  for  which  primary  data  is  necessary,  and  to  ana¬ 
lyzing  and  drawing  conclusions  from  gathered  data.  The  next 
chapter  will  summarize  pertinent  information  from  the 
literature  concerning  trends  in  HIS,  quality  information 
systems,  staff  prerequisites,  and  available  staffing 
sources.  It  will  also  report  the  standards  and  goals  for 
Air  Force  medical  system  offices  as  outlined  in  Air  Force 
Regulation  168-4,  Chapter  14.  Chapter  III  will  describe 
the  survey  instruments  and  methods  of  validation,  popula¬ 
tions  for  each  phase  of  the  research,  data  collection  proc¬ 
ess,  and  methods  of  data  analysis. 

After  all  data  has  been  collected,  statistical  analysis 
of  the  data  will  begin.  Chapter  IV  will  describe  the  analy¬ 
sis  procedures  as  well  as  draw  conclusions  from  the  analysis 
that  serve  to  answer  the  remaining  investigative  questions. 
Finally,  Chapter  V  will  discuss  the  findings  from  the  previ¬ 
ous  chapter  and  answer  the  specific  research  question. 
Moreover,  the  researchers  will  make  recommendations  for 
further  research  on  this  and  related  topics. 
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II:  Literature  Review 


Introduction 

The  Air  Force  Medical  Service  is  presently  exploring  a 
variety  of  sources  for  information  systems  specialists  at 
its  six  medical  centers.  At  least  three  different  sources 
are  currently  being  used,  and  at  least  one  medical  center 
chief  information  system  officer  has  raised  the  question, 
"Which  staffing  method  is  best”  (Obuchowski , 1992 ; 

Sinai  tz,  1991)? 

In  order  to  answer  this  question,  several  issues  were 
explored  in  the  literature.  This  literature  review  will 
answer  the  researchers'  second  investigative  question  by 
summarizing  published  works  on  current  trends  in  hospital 
information  systems.  This  part  of  the  review  will  serve  to 
emphasize  the  importance  of  the  information  system  to  the 
mission  of  the  entire  medical  center  and  the  need  for 
attaining  and  maintaining  currency  and  competitiveness  in 
computer  technology. 

Second,  a  general  definition  of  quality  and  how  it  per¬ 
tains  to  hospital  information  systems  will  be  discussed.  In 
the  process,  definitive  measures  of  quality  in  information 
systems  (ISs)  will  be  explored  in  order  to  establish  guide¬ 
lines  for  gauging  how  well  Air  Force  Medical  Centers  are 
producing  information. 
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Third,  the  researchers  will  summarize  characteristics  o£ 
information  systems  specialists  required  to  maintain  a 
quality  information  system.  These  characteristics  will  also 
be  used  as  subjects  of  discussion  later  in  this  research 
paper.  After  all  data  concerning  each  IS  staff  has  been 
gathered  and  analyzed,  the  researchers  will  discuss  the 
characteristics'  goodness  of  fit  to  the  study  population. 

Finally,  published  viewpoints  on  the  advantages  and 
disadvantages  of  various  staffing  sources  will  be  discussed. 
The  literature  abounds  in  differing  views  and  little  consen¬ 
sus  is  apparent.  This  discussion  will  be  of  interest  after 
the  final  data  analysis  has  been  completed  concerning  the 
relationship  between  the  staffing  sources  of  sample  medical 
systems  offices  and  the  IS  quality  ratings. 

Current  Trends  in  Hospital  Information  Systems 

The  1970s,  80s,  and  90s  have  signaled  the  coming  of  age 
of  the  computer.  In  the  late  1950s,  a  computer  capable  of 
storing  one  megabyte  of  memory  would  have  occupied  about  400 
square  feet,  and  one  with  the  same  memory  capacity  in  the 
mid  1970s  was  less  than  3  square  feet  (Lemon  and  Toole, 
1987:62).  Although  size  hasn't  changed  much,  today's  so 
called  "minicomputers"  can  store  millions  of  megabytes  of 
memory.  In  order  to  survive  in  this  modern,  rapidly  advanc¬ 
ing  electronic  era,  highly  competitive  industries  have  had 
to  aggressively  develop  state-of-the-art  information  systems 
(Lemon  and  Toole, 1987). 
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In  contrast,  the  medical  care  industry  has,  histori¬ 
cally,  made  a  very  poor  showing  in  the  world  of  information 
systems  development  (Lemon  and  Toole, 1987)  because,  prior  to 
about  1984,  hospitals  did  not  have  to  compete  with  one 
another  as  other  industries  did.  Hospitals  charged  patients 
and  insurance  companies  whatever  it  cost  them  to  treat  the 
patient,  plus  a  little  to  ensure  a  profit.  Although  medical 
professionals  did  make  a  feeble  effort  to  automate  some 
functions,  they  did  so  because  computers  were  "nice  to 
have,"  and  hospital  staff  members  strove  to  establish  repu¬ 
tations  as  pioneers  among  their  peers  (Morris, 1986) . 

In  the  last  seven  or  eight  years,  the  situation  has 
changed  dramatically.  Federal  and  state  governments,  along 
with  the  insurance  industry,  have  placed  strict  ceilings  on 
reimbursements  to  hospitals,  and  patients  have  changed  their 
attitudes  concerning  their  expectations  of  the  medical  care 
system.  These  new  developments  have  forced  hospitals  to 
compete  fiercely  with  each  other  for  shrinking  economic 
gains.  Necessarily,  hospital  information  systems  (HIS), 
defined  as  groups  of  computers  and/or  software  designed  to 
support  the  flow  of  interdepartmental  information  within 
hospitals  (Abrami  and  Sneider , 1985 : 44) ,  have  become  the 
primary  weapon  in  the  medical  center  "fight  for  corporate 
survival"  (Morris, 1986: 32 ) . 
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Changes  in  the  Envi ronment . 


The  Economic  Environment.  This  section  discusses 
the  dramatic  changes  in  the  economic,  socio-political,  and 
technological  environments  in  which  hospitals  operate  and 
how  our  medical  care  industry  has  sought  to  adapt  to  those 
changes  through  innovative  information  systems  development. 

In  the  early  1980s,  our  state  and  federal  governments 
and  health  insurance  companies  began  to  realize  that,  with¬ 
out  external  influence,  the  cost  of  health  care  would  con¬ 
tinue  to  skyrocket  as  it  did  in  previous  decades.  They 
began  to  implement  controls  to  curb  costs  through  prospec¬ 
tive  payment  systems  and  diagnosis  related  groups  (DRGs). 
Simply  put,  these  plans  dictated  the  amount  of  monetary 
reimbursement  according  to  the  patient's  diagnosis,  no 
matter  how  much  it  cost  the  hospital  to  provide  treatment 
(Kim  and  Michelman,1990) . 

Hospital  administrators  found  themselves  having  to  make 
a  very  rapid  transition  from  working  in  a  noncompetitive 
industry  whose  only  function  was  to  fulfill  a  social  need  to 
one  based  on  business  and  competition.  Patients  began 
shopping  around  for  the  hospital  that  would  completely 
provide  for  their  medical  needs  at  a  price  their  insurance 
company  was  willing  to  pay  (Morris , 1986) . 

Concurrent  with  the  imposition  of  fixed  fee  payment 
structures  on  civilian  hospitals,  the  public  and  the  federal 
government  were  placing  severe  cost  constraints  on  military 
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hospitals  as  well.  The  public  began  to  balk  at  overcharging 
on  the  part  of  government  contractors  and  the  apparent  waste 
of  taxpayer  money.  Simultaneously,  Congress  began  making 
drastic  cuts  in  Department  of  Defense  spending.  Hence, 
military  hospitals  were  forced  to  live  within  their  yearly 
budgets  with  little  hope  of  more  money  when  they  overspent 
(Shaw, 1984) . 

Thus,  in  both  the  civilian  and  military  sectors,  medical 
cost  managers  and  administrators  were  forced  to  look  outside 
their  industry  for  new  approaches  to  get  the  "most  bang  for 
the  buck."  Most  are  finding  the  answers  to  their  survival 
in  the  use  of  information  technology  (Hofman,1984;  Lemon  and 
Toole, 1987 ) . 

The  Socio-Political  Environment.  At  the  same  time 
medical  care  costs  have  come  under  intense  scrutiny,  both 
hospital  employees  and  the  general  public  have  changed  their 
attitudes  toward  the  way  they  provide  and  receive  medical 
treatment.  On  the  one  hand,  patients  have  begun  to  view 
themselves  as  customers  seeking  products  and  services  from 
the  health  care  system.  They  are  no  longer  willing  to 
endure  poor  customer  service  and  are  quick  to  switch  hospi¬ 
tals  and  care  providers  if  they  are  not  satisfied.  Custom¬ 
ers  now  expect  short  waiting  lines,  immediate  appointments, 
and  rapid,  efficient  medical  records  processing.  Thus, 
medical  care  facilities  are  forced  to  compete  with  each 
other  in  cost  of  treatment  and  quality  of  the  customer 
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service  they  provide.  Besides  financial  management, 
information  systems  have  become  invaluable  in  performing 
other  f unctions --scheduling  patients,  reporting  test 
results,  and  distributing  other  patient  information 
(Hofman,1984;  Kowalski , 1991) . 

On  the  other  hand,  while  demanding  that  less  time  be 
taken  away  from  direct  patient  care,  hospital  employees  have 
generally  become  more  computer  literate.  Since  computers 
are  prevalent  in  homes  and  schools,  staff  members  now,  more 
than  ever,  accept  the  use  of  technology  as  a  means  of 
improving  their  on-the-job  productivity.  Thus,  the  hospi¬ 
tal's  employees  have  become  the  information  systems  depart¬ 
ment's  internal  customers.  The  result  is  even  greater 
pressure  to  improve  hospital  information  systems  (Lemon  and 
Toole, 1987) . 

In  contrast  to  the  above  factors  encouraging  hospitals 
to  develop  more  modern  information  systems,  there  are  other 
socio-political  challenges  within  those  hospitals.  The 
first  challenge  is  to  lessen  the  fears  of  the  small  group 
existing  within  every  organization  which  suffers  from 
"computerphobia"  ( Sehr , 1985 : 78 ) .  These  individuals  are 
highly  resistant  to  change  and  even  view  computers  as  a 
threat  to  their  position  within  the  organization.  Many  feel 
their  jobs  may  be  eliminated  through  automation  (Mann, 1988). 

Further,  many  physicians  feel  threatened  by  computeriza¬ 
tion.  They  feel  they  have  exclusive  rights  to  much  of  the 
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patient  information  they  collect  and  that  it  should  not  be 
made  available  through  hospital -wide  information  systems. 

In  addition,  many  physicians  view  computers  as  "'control 
systems'  whose  sole  purpose  is  to  monitor  and  report  on 
their  behavior"  (Kim  and  Michelman,1990;  204).  The  result 
has  been  a  long-fought  battle  between  physicians  and  hospi¬ 
tal  administrators  on  the  subject  of  technological  advance¬ 
ment  of  hospital  information  systems  (Kim  and 
Michelman, 1990 ) . 

Hence,  hospital  information  systems  experts  and  adminis¬ 
trators  are  being  pulled  in  two  directions  in  their  efforts 
to  more  effectively  use  their  computer  systems  to  survive  in 
the  world  of  business.  While  trying  to  keep  "o  with  rapidly 
advancing  technology,  they  must  also  make  every  effort  to 
appease  those  social  groups  adamantly  opposed  to  computer 
innovation. 

The  Technological  Environment.  As  tight  budget 
constraints,  demands  for  quality  customer  service,  and 
demands  for  high  productivity  are  on  the  rise  in  the  medical 
care  industry,  the  world  of  automation  is  evolving  at  a 
rapid  pace.  Many  of  these  breakthroughs  significantly 
broaden  the  possibilities  in  expanding  today's  hospital 
information  systems.  A  few  of  the  most  consequential  devel¬ 
opments  which  are  directly  relevant  to  the  current  study  are 
discussed  here. 
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One  of  the  most  important  developments  in  the  world  of 
computing  over  the  last  decade  has  been  the  decreasing  cost 
of  hardware  (by  about  20  percent  per  year) .  This  price 
reduction  has  enabled  hospital  administrators  to  consider 
new  systems  as  a  viable  alternative  (Lemon  and 
Toole, 1987:62) . 

Further,  a3  noted  earlier,  microchip  technology  has 
decreased  the  size  of  computers  and  increased  memory  capa¬ 
city,  thereby  adding  capabilities  beyond  the  imagination 
of  the  1950s  pioneers.  Memory  storage  capacities  are  a 
million  times  greater  and  mainframes  are  a  thousand  times 
smaller  than  those  of  only  three  decades  ago.  Moreover, 
machines  are  now  able  to  process  information  more  than  a 
hundred  times  faster  than  they  could  in  1970  (Lemon  and 
Toole, 1987:62) . 

Finally,  improved  telecommunications  have  greatly  af¬ 
fected  computerization.  The  extent  to  which  this  new  tech¬ 
nology  has  improved  information  distribution  is  well-stated 
by  Lemon  and  Toole: 

Major  changes  in  the  communications  industry  have 
occurred  in  recent  years,  largely  due  to  deregu¬ 
lation  of  the  telephone  industry.  Historically, 
communications  has  [sic]  been  a  limiting  and  expen¬ 
sive  link  in  transferring  large  amounts  of  digital 
data.  Information  for  computers  had  to  be  converted 
to  analog  form  before  it  could  be  transmitted,  then 
reconverted  to  digital  pulses  at  the  destination. 

This  situation  is  changing  rapidly  as  dedicated 
transmission  links  and  related  software  are  being 
developed.  (Lemon  and  Toole, 1987: 62-63) 
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Lemon  and  Toole  go  on  to  that  suggest  fiber-optic  lines  will 
enable  information  to  be  transmitted  at  "ultra-high  speeds" 
(Lemon  and  Toole,1987 : 63) . 

Maintaining  a  competitive  edge  in  the  world  of  business 
through  information  systems  has  proven  to  be  a  nearly  over¬ 
whelming  challenge  for  industries  that  began  automating 
decades  ago.  Hospital  staffs  are  being  challenged  even  more 
in  their  attempts  to  narrow  the  10  to  20  year  information 
technology  gap  between  the  medical  care  industry  and  other 
industries . 

Meeting  the  Challenge.  New  applications  for  hospital 
information  systems  that  help  medical  centers  adapt  to  the 
changing  economic  and  socio-political  environments  are 
rapidly  becoming  available.  They  can  be  grouped  into  three 
broad  categories:  financial  management,  patient  care,  and 
decision  support  and  strategic  planning. 

Pinancial  Management .  One  of  the  largest  items  in 
any  hospital’s  budget  is  its  supplies  and  the  movement  of 
those  supplies.  Wagner  states  that  medical  care  facilities 
will  spend  approximately  16  to  28  percent  of  their  annual 
budgets  on  supply  purchases.  And  surveys  have  shown  that 
for  every  $1  spent  to  purchase  an  item,  an  additional  $0.70 
to  $1  is  spent  on  getting  the  item  to  its  final  destination, 
the  user  (Wagner , 1990 : 23 ) .  The  combination  of  supply  pur¬ 
chases  and  supply  movement  can  amount  to  as  much  as  45%  of 
the  total  budget  (Mendenhal 1 , 1988 : 54 ) .  Obviously,  hospital 
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administrators  have  found  the  area  of  materiel  management 
ripe  with  potential  for  trimming  costs  through  information 
systems  innovation. 

Two  new  concepts  in  controlling  inventory  are  slowly 
being  accepted  by  the  medical  field.  Both  methods  integrate 
new  information  technology  into  supply  management.  The 
first,  "just-in-time  inventory,"  requires  "smaller  but  more 
frequent  bulk  deliveries  to  the  hospital,  thereby  reducing 
stockpiles  of  supplies  stored  at  the  hospital"  (Wagner, 
1990:23) . 

The  second,  called  "stockless  inventory,"  entails  no 
bulk  deliveries,  but  instead  requires  suppliers  to  "send 
supplies  already  packaged  for  end-user  sites.  They  may  even 
deliver  the  supplies  directly  to  those  sites  throughout  the 
hospital"  (Wagner, 1990:23) .  Both  systems  require  current, 
immediate  communications  between  the  hospital  and  the  dis¬ 
tributor.  Often,  end-user  sites  (wards  and  clinics)  have 
direct  on-line  access  to  the  distributor's  computers  through 
barcode  scanners,  enabling  them  to  order  their  own  supplies 
and  eliminating  the  use  of  middlemen  in  the  hospital's 
materials  management  department  (Wagner , 1990 ) . 

Patient  Care.  One  of  the  most  intriguing  new 
concepts  in  hospital  automation  is  called  point-of -care  or 
bedside  computing.  Terminals  placed  at  the  patient's  bed¬ 
side  speed  up  the  charting  process  by  nursing  personnel  and 
the  querying  and  ordering  process  by  physicians.  Nurses 
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eliminate  about  one  hour  per  shift  previously  spent  on 
paperwork,  allowing  them  more  time  for  direct  patient  care 
(Lemon  and  Toole, 1987 : 66) .  Thus,  productivity  and  job 
satisfaction  are  enhanced  and  the  hospital  saves  money. 
"Although  still  largely  in  the  research  and  development 
stage,  point-of -care  computing  is  projected  by  many  to  hold 
the  strongest  potential  for  achieving  real  productivity 
gains  through  information  technology"  (Lemon  and  Toole, 
1987:66) . 

Further,  complete  automation  of  the  patient's  chart  by 
the  caregivers  provides  the  hospital  with  the  "ultimate 
database"  (Mendenhall ,1988:54) .  The  computerized  record  of 
the  patient's  hospital  stay  can  now  become  the  source  docu¬ 
ment  for  quality  assurance  review,  the  ordering  of  tests  and 
recording  of  results,  and  even  information  pertinent  to  the 
patient's  bill  (Mendenhal 1 , 1988 ) . 

Decision  Support  and  Strategic  Planning.  Perhaps 
the  new  information  system  technology  that  provides  the 
greatest  long-term  benefit  is  in  the  area  of  decision  sup¬ 
port  to  hospital  administrators.  Decision  support  systems 
(DSSs)  can  aid  in  long  range  planning  and  forecasting  future 
cost  savings  and  improvements  in  customer  service  and  pro¬ 
ductivity.  These  systems  enable  decision  makers  to  extract 
and  summarize  pertinent  information  from  computerized 
patient  records,  materiel  management  accounts,  admission/ 
discharge  scheduling  records,  and  billing  accounts. 
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Further,  they  will  allow  access  to  information  concerning 
market  share  in  relation  to  other  hospitals  in  the  same 
geographic  area  through  public  data  bases  (Lemon  and 
Toole,1987;  Kim  and  Michelman, 1990 ;  Hard, 1990b). 

The  hospital  administrator's  goal  in  implementing  deci¬ 
sion  support  systems  is  to  develop  a  tool  to  help  him  or  her 
make  quality  strategic  decisions  concerning  the  hospital's 
future  direction.  A  good  DSS  requires  total  integration  of 
all  computer  systems  within  the  organization,  to  include 
both  historical  and  current  data  and  future  requirements,  so 
that  the  decision  maker  has  all  the  information  he  or  she 
needs  to  make  quality  decisions  (Lemon  and  Toole, 1987;  Kim’ 
and  Michelman, 1990;  Hard, 1990b). 

DSSs  can  also  be  valuable  to  medical  providers.  They 
can  "assist  in  diagnosing  and  treating  illnesses,  selecting 
antibiotics  for  infectious  diseases,  and  managing  complex 
chemotherapy  for  cancer  patients"  (Lemon  and  Toole, 1987: 64) . 

If  users  are  unhappy  with  the  information  system,  they 
will  not  use  it  unless  forced  to  do  so,  and  the  system  will 
not,  in  all  likelihood,  live  up  to  management's  expecta¬ 
tions.  Thus,  in  implementing  new  applications,  managers 
must  always  keep  in  mind  the  social  and  political  ramifica¬ 
tions  discussed  earlier. 

First,  system  designers  must  always  remember  that  one  of 
the  critical  success  factors  of  today's  hospitals  is  quality 
customer  service.  If  the  introduction  of  a  new  computer 
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technology  does  not  have  the  potential  to  live  up  to  this 
demand,  management  may  want  to  give  higher  priority  to 
another  technology  (Kowalski , 1991 ) . 

Further,  the  needs  of  the  hospital  staff  are  also  impor¬ 
tant.  In  addition  to  bedside  computing,  which  increases 
productivity  of  nurses,  other  new  ideas  are  currently  being 
tried  to  increase  job  satisfaction.  One  of  these  ideas  is 
the  concept  of  an  information  center.  Besides  providing 
on-going  training,  this  department's  job  entails  convincing 
hospital  employees  of  the  need  for  change,  hotline  IS  sup¬ 
port,  and  maintaining  a  resource  library.  The  goal  in 
providing  these  services  is  to  change  the  attitudes  of  those 
fearing  computers  so  that  they  become  comfortable  with 
information  processing  and  realize  help  is  always  available 
(Sockolow,1989) . 

To  placate  the  anxieties  of  physicians,  the  information 
center  staff  must  be  very  helpful  in  developing  specialized 
applications  tailored  to  the  needs  of  individuals.  End-user 
computing  allows  physicians  to  determine  their  needs  in  an 
information  system,  and  thus  reduce  the  feelings  of  being 
controlled  by  the  system  (Kim  and  Michelman, 1990 ) . 

An  overview  of  the  hospital  information  system  develop¬ 
ment  process  is  shown  in  Figure  1.  Hospitals  are  pushed 
toward  the  implementation  of  more  comprehensive  systems  by 
growing  budget  constraints,  the  advent  of  new  computer 
technology,  and  increasing  computer  literacy  on  the  part  of 
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hospital  employees.  Meanwhile,  pockets  of  resistance,  in 
the  form  of  staff  computerphobia  and  fear  of  territorial 
encroachment  among  physicians,  tend  to  slow  the  change  to 
comprehensive  automation. 

After  hospital  managers  have  determined  that  the  poten¬ 
tial  benefits  of  system  development  outweigh  the  risks  of 
furthering  job  dissatisfaction  among  the  dissenters,  they 
generally  install  new  applications  fitting  into  one  of  three 


Figure  1.  HIS  Development  Influences  and  New  Applications 
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categories.  These  application  categories  are  patient  care, 
decision  support  and  strategic  planning,  and  financial 
management.  The  information  system  staff  seeks  to  fulfill 
two  functions:  overseeing  the  transition  and  convincing 
those  resistant  to  change  of  the  potential  rewards. 

The  1980s  brought  a  new  business-oriented  climate  to 
hospitals.  Along  with  the  new  climate,  the  last  decade 
brought  the  coming  of  age  of  the  computer.  Information 
systems  have  matured  into  vital  competitive  weapons  in 
effecting  necessary  changes  in  hospital  business  strategy. 

How  can  one  measure  the  HIS's  level  of  success  in  help¬ 
ing  the  hospital  meet  its  strategic  goals?  How  can  one 
measure  the  HIS's  level  of  success  in  meeting  the  needs  and 
desires  of  its  customers?  The  answer  to  these  questions 
lies  in  the  measurement  of  the  information  system's  most 
critical  success  factor--its  quality. 

What  is  Quality  in  Information  Systems? 

Kane  ga  naru  ka  ya 
Shumoku  ga  naru  ka 
Kane  to  shumoku  no  ai  ga  naru 

Is  it  the  bell  that  rings. 

Is  it  the  hammer  that  rings. 

Or  is  it  the  meeting  of  the  two  that  rings? 
(Barankin: 216,1964) 

Measuring  quality  in  the  production  of  goods  and  serv¬ 
ices  has  become  a  science  in  and  of  itself  in  recent  years. 
Many  different  factors  must  contribute  to  this  measurement. 
In  the  absence  of  even  one  of  the  factors,  the  quality 
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"bell"  will  not  ring.  In  this  section,  the  most  prominent 
views  of  the  measurement  of  quality  will  be  discussed. 

To  adequately  assess  the  level  of  quality  of  an  informa¬ 
tion  system  one  must  first  precisely  define  the  word 
"quality."  This  is  something  many  researchers,  philoso¬ 
phers,  and  managers  have  been  attempting  to  do  for  decades. 
Quality  is  a  slippery  term  whose  meaning  has  been  evolving 
since  the  dawn  of  the  manufacturing  age  ( Juran, 1989) . 

Immediately  following  World  War  II,  the  United  States 
found  itself  in  a  seller's  market  with  the  rest  of  the 
world.  All  of  its  factories,  left  unscathed  by  the  ravages 
of  battle,  produced  goods 'at  breakneck  speed  in  order  to 
keep  up  with  demand.  Quality  control  was  of  little  value 
since  customers  were  satisfied  with  whatever  they  could  get 
( Juran, 1989:6-7) . 

At  the  same  time  that  our  country  was  lengthening  its 
lead  among  manufacturing  giants,  the  Japanese  set  their 
sights  on  entering  the  race.  W.  Edwards  Deming  was  among 
the  first  experts  on  quality  to  assist  Japan  in  this 
endeavor.  He  began  by  introducing  that  country's  manufac¬ 
turers  to  his  12  points  for  the  transformation  of  industry. 
His  aim  was  to  steer  management  away  from  profit  goals, 
management  by  objective,  and  quotas.  He  advocated  a  "con¬ 
stancy  of  purpose  toward  improvement  of  product  and  service" 
(Deming, 1982:3) .  His  original  12  points  expanded  to  14 
after  his  first  work  with  the  Japanese  (Deming, 1982 ) . 
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Through  the  assistance  of  Deming  and  others  and  the 
grass-roots  determination  of  the  Japanese  people,  many 
products  manufactured  today  in  Japan  are  preferred  by  cus¬ 
tomers  the  world  over  while  many  American  industries  are 
floundering  (Garvin , 1988 ;  Deming , 1982 ) . 

Deming  states: 

With  the  storehouse  of  skills  and  knowledge  contained 
in  its  millions  of  unemployed,  and  with  the  even  more 
appalling  underuse,  misuse,  and  abuse  of  skills  and 
knowledge  in  the  army  of  employed  people  in  all  ranks 
in  all  industries,  the  United  States  may  be  today  the 
most  underdeveloped  nation  in  the  world.  (Deming, 1982 : 6) 

For  the  last  two  decades,  American  producers  have  gradu¬ 
ally  begun  to  realize  old  ways  will  no  longer  keep  our 
products  and  service  competitive  at  home  or  in  the  world’s 
marketplace  (Garvin, 1988 ) .  The  emphasis  has  begun  to  shift 
from  maintaining  the  bottom  line  to  producing  what  the 
customer  wants.  David  Garvin  states: 

Quality  is  fast  becoming  one  of  the  competitive 
issues  of  the  1980s  and  1990s.  A  wave  of  imports, 
federal  and  state  programs,  and  increased  customer 
sensitivity  have  combined  to  give  it  new  visibility. 
Pressures  for  improvement  have  become  intense.  The 
result  is  a  heightened  interest  in  quality  management 
at  many  U.S.  companies  and  a  growing  recognition  of 
quality's  strategic  importance.  (Garvin, 1988 : xi ) 

What  is  Quality?  Having  fully  realized  the  impact  of 

quality  output  on  business  success,  managers  next  sought  to 

define  quality  in  terms  that  could  be  quantified  at  any 

point  in  time  in  the  business  cycle  and  improvement  that 

could  be  measured.  Useful  definitions  are  abundant  in 
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recent  literature  and  all  are  derived  from  the  same  two 
themes:  industry  standards  and  customer  desires. 

Ernst  and  Young  summarize  the  incorporation  of  standards 
and  desires  very  clearly.  They  say  the  first  step  toward 
achieving  product  quality  is  "conf otmance  to  standards,"  or 
production  free  of  flaws.  Next  is  "meeting  customer 
requirements,"  or  the  degree  to  which  the  user  is  satisfied 
with  the  product's  design  and  usefulness  (Ernst  and 
Young, 1990 : 4) . 

High  quality  will  not  be  achieved  by  only  satisfying  one 
of  the  requirements.  In  fact,  conformance  to  industry 
specifications  only  indicates  the  product  is  not  of  poor 
quality.  Neither  is  a  high  level  of  quality  ensured--only 
neutrality.  Only  when  the  second  requirement,  that  of 
satisfying  the  customer,  is  met  does  the  product  have  a 
positive  quality  value  (Ernst  and  Young, 1990 : 4-5 ) .  This 
two-portion  concept  of  quality  production  is  illustrated  in 
Figure  2,  on  the  following  page. 

J.M.  Juran's  term  "fitness  for  use"  aptly  describes 
Figure  2.  Conformance  to  specifications  involves  measure¬ 
ments  such  as  error  rates,  waste  and  rework,  time  to  market, 
and  field  failures.  The  standards  for  each  of  these  meas¬ 
urements  are  set  by  people  within  the  corporation  and  usu¬ 
ally  do  not  take  into  account  the  desires  of  the  cus¬ 
tomer.  These  measurements  are  not  a  new  idea.  They  have 
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been  among  the  most  heavily  relied  upon  production  quality 
control  parameters  for  decades  ( Juran , 1989 : 15-16 ) . 


HIGH 

QUALITY 


mutual 

QUALITY 

LOU 

QUALITY 

Figure  2.  Measuring  Quality  (adapted  from  Ernst  and 
Young, 1990) 

The  measurements  involved  in  determining  the  value  of 
the  right  side  of  the  above  illustration  (customer  satisfac 
tion)  are  the  new  ingredients  in  the  total  quality  gauge. 

As  American  business  people  realize  that  competition  is  the 
key  to  survival,  they  are  also  realizing  that  the  customer 
is  the  ultimate  decision  maker  in  determining  a  product's 
value.  Ernst  and  Young  state  that  "while  quality  of  con¬ 
formance  is  of  obvious  importance,  conformance  alone  does 
not  ensure  competitiveness.  One  can  easily  imagine  a  per¬ 
fect  (defect-free)  product  that  no  one  wants"  (Ernst  and 


Young , 1990 : 4-5 ) .  Indeed,  an  unsold,  though  flawless, 
product  does  little  to  improve  a  corporation's  financial 
standing.  Further,  the  former  Soviet  manufacturing  system 
was  a  good  example  of  how  products  can  meet  preset  specifi¬ 
cations  without  being  of  high  quality.  State  standards 
abounded  but  customer  preferences  were  never  considered. 

The  resulting  goods  were  of  questionable  quality  and,  if  put 
on  the  market  in  a  competitive  business  environment,  would 
have  probably  only  gathered  dust  (Forker, 1991) . 

The  new  movement  toward  attempting  to  fulfill  customer 
desires  has  led  to  methods  of  precisely  measuring  a 
product's  effectiveness  in  .fulfil  ling  those  desires.  Garvin 
is  responsible  for  authoring  a  framework  for  product  quality 
analysis  which  includes  eight  dimensions  or  categories  of 
quality: 

1.  Performance  -  key  operational  attributes 

2.  Features  -  "nice  to  have"  options  that  enhance  the 
product's  functionality 

3.  Reliability  -  measure  of  a  product's  probability 
of  failure 

4.  Conformance  -  degree  to  which  a  product  meets  the 
manufacturer's  specifications 

5.  Durability  -  length  of  a  product's  life 

6.  Serviceability  -  speed  and  ease  of  repair  and  the 
degree  the  manufacturer  is  willing  to  provide 
prompt,  knowledgeable,  and  courteous  service 

7.  Aesthetics  -  a  very  subjective  measure  of  the  degree 
a  product  appeals  to  the  customer’s  senses 
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8.  Perceived  Quality  -  reputation  and  affiliation  of 
the  producer;  often  a  product  will  be  purchased  by 
virtue  of  this  dimension  alone  (Garvin, 1988 : 49-60 ) 

With  the  exception  of  conformance,  the  best  means  of  measur¬ 
ing  each  of  the  dimensions  include  assessing  the  products  of 
competitors  and  surveying  potential  buyers  "to  establish 
what  customers  mean  when  they  say  one  product  is  of  higher 
quality  than  another"  (Garvin, 1988 : 24) . 

Other  authors  have  identified  their  own  sets  of  quality 
indicators,  but  all  direct  managers  to  rely  heavily  on  the 
customer's  perception  as  the  ultimate  determinant  of  quality 
(Emmelhainz , 1991 ;  Juran,1989;  Murray, 1991;  Spitzer , 1991 ) . 

After  years  of  poor  performance  on  the  part  of  American 
indust.;’,  quality  management  goals  have  finally  become  top 
priorities  among  our  most  competitive  companies.  In  fact, 
striving  to  be  "world  class"  has  become  the  guiding  princi¬ 
ple  in  many  strategic  planning  circles  (Murray : 7 , 19S1 ) .  The 
1992  Malcolm  Baldrige  National  Quality  Award  will  be  given 
to  the  company  which  best  meets  the  criteria  in  seven  cate¬ 
gories:  "leadership,  information  and  analysis,  strategic 
quality  planning,  human  resources,  quality  assurance, 
results,  and  customer  satisfaction"  ( Sonnenberg , 1990 : 63 ) . 

These  criteria  have  been  so  successful  in  truly  measur¬ 
ing  quality  that  "30,000  [companies]  are  using  the  criteria 
to  assess  their  own  performance,  to  guide  quality  planning, 
and  to  understand  the  components  of  a  total  quality  program" 
(Sonnenberg, 1990 : 63) .  This  widespread  interest  in  quality 
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improvement  throughout  the  country  is,  indeed,  a  long-needed 
trend  in  the  way  our  producers  do  business. 

Hence,  to  answer  the  question,  "What  is  quality?",  a 
product's  quality  is  the  degree  to  which  the  product  meets 
industry  standards  and  specifications  and  the  degree  to 
which  the  product  satisfies  the  needs  and  desires  of  the 
customer  (Deming, 1982 ;  Ernst  and  Young, 1990; 

Juran,1989) . 

Applying  Quality  to  Hospital  Information  Systems.  The 
need  to  provide  better  quality  service  in  our  nation's 
medical  industry  resulted  from  governmental  and  insurance 
industry  pressure  to  lower  the  cost  of  care  and  from  demands 
for  increased  customer  satsifaction.  As  was  discussed 
earlier,  a  hospital's  information  system  is  a  key  element  in 
meeting  this  need.  Further,  the  information  system  itself 
must  have  a  high  level  of  quality  in  order  to  contribute  to 
the  entire  organization's  level  of  quality.  Thus,  quantita¬ 
tive  measures  of  the  system's  quality  must  be  made  to  assess 
its  worth  to  the  hospital  ( Spitzer , 1991 ) . 

Spitzer  has  recommended  measuring  the  extent  to 
which: 

-  the  customer-desired  outc  ->s  are  achieved  (and  this 
is  realized  by  the  custom*  ' 

-  undesired  outcomes  and  repercussions  are  minimized, 
and 

-  the  service  is  a  positive  experience  for  the 
customer.  (Spitzer, 1991: 24-25) 
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Thus,  even  in  a  service-only  discipline  such  as  information 
systems,  the  same  key  indicators  of  quality  as  those  used  in 
manufacturing  should  be  applied.  Murray  reinforces  this 
thinking  by  saying,  "Defining  world  class  [information 
system  performance]  starts  with  seeing  the  world  from  the 
customer's  point  of  view"  (Murray , 1991 : 7 ) .  He  goes  on  to 
describe  his  "eight  steps  to  target  and  develop  the  business 
case  for  quality  improvement  [in  information  systems]" 
(Murray , 1991 : 8 ) .  Not  surprisingly,  these  "eight  steps" 
almost  exactly  mirror  Garvin's  (1988)  "eight  dimensions" 
mentioned  earlier.  Therefore,  gauges  determining  the 
exient  to  which  (1)  an  information  system  conforms  to  indus¬ 
try  standards  and  specifications,  and  (2)  an  information 
system  meets  the  needs  and  desires  of  its  users  are  appro¬ 
priate  in  measuring  the  HIS's  level  of  quality. 

People  Who  Manage  Quality  Hospital  Information  Systems 
With  the  drastic  changes  currently  being  made  in  the 
tactics  of  business  survival,  every  person  in  the  organiza¬ 
tion  must  also  adapt  to  the  new  work  environment.  In  the 
case  of  information  systems,  people  are  the  key  element  at 
the  ends  of  the  process,  that  of  supplying  the  right  kind  of 
information,  in  an  adequate  amount,  at  the  right  time  in 
order  to  meet  the  goals  of  the  organization.  People  (infor¬ 
mation  system  specialists)  must  be  able  to  interpret  the 
information  needs  of  other  people  (system  users)  and 
implement  processes  that  meet  those  needs.  Hence,  great 
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emphasis  must  be  placed  on  developing  a  quality  information 
system  staff  conscious  of  every  aspect  involved  in  producing 
quality  information  (Moad,1990). 

How  would  one  go  about  describing  the  ideal  information 
system  staff  member?  The  next  section  will  summarize  the 
views  of  several  experts  in  information  system  management. 

T«»r;hnn logical  Skills.  A  high  level  of  technological 
competence  is  a  mandatory  requirement  for  every  member  of 
the  IS  department.  As  was  discussed  in  a  previous  section, 
the  pace  at  which  computer  technology  is  advancing  is  stag¬ 
gering.  The  ability  of  an  organization  to  use  these 
advances  to  keep  one  step  ahead  of  the  fierce  competition  is 
fundamental  to  the  organization’s  survival  (Moad,1990). 

Information  technicians  who  are  lagging  behind  the  times 
may  find  themselves  dwelling  on  main-frame  orientation, 
analysis,  and  project  management,  when  they  should  be  focus¬ 
ing  on  state-of-the-art  technologies  such  as  personal  com¬ 
puters,  local  area  networks,  and  fourth  generation  lan¬ 
guages.  Constant  training  and  updating  are  mandatory  for 
every  IS  person.  If  on-the-job  training  is  not  offered  to 
them,  they  must  actively  seek  out  programs  during  off-duty 
time  to  keep  abreast  of  new  technology  (Moad,1990). 

Additionally,  the  day  of  the  specialist  is  over  in 
information  systems.  Because  of  the  high  degree  of  integra¬ 
tion  across  the  various  systems  in  the  organization, 
networks  and  data  base  sharing  require  the  IS  workers  to  be 
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experts  in  an  increasingly  wide  array  of  information-related 
subjects  (Moad,1990). 

The  task  of  keeping  a  staff  versed  on  technology  issues 
is  a  challenge  by  itself  to  the  IS  department  manager.  The 
manager's  job  becomes  even  greater  when  two  other  skill 
areas--interdepartmental  relations  and  business  acumen--are 
added  to  the  list  of  employee  requirements. 

Organizational  Public  Relations.  Coupled  with  the 
requirement  for  keeping  up  with  a  constantly  changing  tech¬ 
nological  environment,  IS  staff  members  must  also  learn  to 
work  face-to-face  with  users.  The  development  of  end-user 
computing  is  driving  the  information  people  out  of  their 
offices  and  computer  rooms  and  into  the  customers'  domains. 
They  must  constantly  keep  in  touch  with  users,  the  jobs  they 
do,  and  the  special  jargon  they  use  in  order  to  anticipate 
needs  (Stokes).  Ouellette  writes  that  this  intimate  inter¬ 
action  with  customers  should  extend  to  the  point  where  IS 
professionals  even  "read  their  clients'  trade  publications 
and  attend  their  clients'  seminars  and  conferences" 
(Ouellette, 1990:353) . 

The  IS  manager  must  ensure  his  or  her  people  are  contin¬ 
ually  abreast  of  changes  in  computing  technology  and  are 
very  familiar  with  the  various  career  specialties  of  the 
clients.  Still,  this  knowledge  is  not  enough  to  guarantee 
quality  among  the  staff.  They  must  also  become 
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knowledgeable  about  the  business  of  the  overall  organization 
(Stokes, 1991) . 

Business  Concepts.  The  third  area  of  expertise  the 
manager  must  nurture  among  his  staff  and  within  himself 
concerns  "the  enterprise’s  vision,  mission,  objectives,  and 
strategies"  ( Stokes , 1991 : 46) .  Only  then,  will  the  entire 
department  totally  grasp  their  reason  for  being  (Stokes, 
1991)  . 

The  cultivation  of  business  acumen  within  the  informa¬ 
tion  department  will  lead  to  an  understanding  of  marketing 
and  sales  strategies,  consulting,  budgeting,  the  organiza¬ 
tional  power  structure,  planning,  and  competitive  position¬ 
ing  ( Stokes, 1991 ) .  When  the  staff  has  a  full  understanding 
of  these  business  concepts  they  will  find  themselves  in  the 
role  of  "strategic  partners"  with  upper  management 
(Ouel lette , 1990 : 354) .  They  will  be  able  to  "have  a  strate¬ 
gic  impact  on  corporate  goals,  and  to  position  themselves 
for  future  success"  (Ouel 1 ette, 1990 : 355 ) . 

Certainly,  the  degree  to  which  each  of  the  above  three 
skill  areas  needs  to  be  developed  in  each  IS  individual  is 
dependent  upon  that  individual’s  level  in  the  management 
hierarchy.  A  person  on  the  lower  rungs  of  the  IS  ladder 
needs  to  place  a  great  deal  of  emphasis  on  technological 
competence,  moderate  emphasis  on  interdepartmental  relations 
skills,  and  little  or  no  concern  with  corporate  business 
strategies.  On  the  other  hand,  the  department  manager  needs 
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to  focus  him/herself  primarily  on  organizational  business 
goals  and  public  relations  skills  while  technology  is  of 
tertiary  importance  (Hard, 1990a;  Stokes , 1991 ) .  Stokes' 
illustration  correlating  the  three  area  of  competence  to 
management  level  is  shown  in  Figure  3,  below. 


Figure  3.  Competency  Development  by  Managerial  Level 
(adapted  from  Stokes, 1991) 

Organizational  Loyalty.  The  final  ingredient  in  the 
formula  for  a  total  quality  information  system  staff  is  not 
a  skill,  but  a  belief.  Numerous  authors  have  written  that 
shifting  loyalty  from  profession  to  organization  is  critical 
to  the  success  of  the  IS  department  (Bowen, 1991; 

Carlyle, 1990;  DeJarnett , 1990 ;  Ouellette, 1990) . 

Devotion  to  the  mission  of  the  corporation  on  the  part 
of  employees  naturally  leads  to  improving  the  human 
relations  and  business  skills  mentioned  above.  Information 
systems  people  begin  to  relate  to  their  customers  and  are 
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more  will..**,  to  get  out  and  see  what  is  going  on  in  the  rest 
of  the  organization.  They  no  longer  think  of  themselves  as 
a  stand-alone  group  doing  a  job  unrelated  to  those  outside 
the  department.  Instead,  they  believe  "we’re  all  in  this 
together."  Most  importantly,  they  have  a  personal  stake  in 
the  success  of  the  whole  company,  not  just  their  own  infor¬ 
mation  systems  (Bowen, 1991;  Carlyle, 1990;  DeJarnett , 1990 ; 
Ouel lette, 1990 ) . 

In  review,  the  consensus  reached  by  most  information 
system  management  authors  is  that  the  most  critical  charac¬ 
teristics  required  of  quality  IS  people  are  loyalty  toward 
the  organization  and  competence  in  technology,  human  rela¬ 
tions,  and  business.  These  traits  instilled  into  each 
individual  member  will  result  in  what  Bowen  terms  "person- 
organization  fit"  and  a  quality  information  system  depart¬ 
ment  (Bowen, 1991:37-38) . 

Sources  of  Information  Systems  Specialists 

Finding  people  who  will  meet  the  formidable  criteria  of 
quality  information  system  specialists  is  proving  to  be 
quite  a  challenge  for  managers.  Until  recent  years,  skilled 
computer  technicians  met  the  organization's  needs,  but  now, 
the  information  system  must  be  piloted  by  people  with  a 
repertoire  of  talents  ranging  from  business  to  public 
relations  to  finely-tuned  technical  expertise.  Where  should 
the  chief  information  officer  turn  to  uncover  the  best 
sources  of  people? 
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Experts  in  the  field  have  fairly  closely  agreed  on  all 
concepts  discussed  in  this  review  of  the  literature  until 


this  point,  but  when  broaching  the  subject  of  staffing 
sources,  the  authors  have  divergent  opinions.  Some  have 
attempted  to  adapt  in-house  employees  to  the  new  quality 
environment  while  others  have  contracted  management  of  the 
information  systems  with  an  outside  source.  The  latter 
method  is  called  "outsourcing"  (Kel 1 y , 1990 : 103 ) .  This 
section  will  cover  the  advantages  and  disadvantages  of  each. 
In  most  instances,  the  advantages  of  one  will  be  the  disad¬ 
vantages  of  the  other  and  vice  versa . 

In-House  Staffing.  Organizational  loyalty  is  the  most 
commonly  cited  advantage  of  using  people  from  within  an 
organization  to  run  the  information  system.  Employees  have 
developed  a  sense  of  belonging  to  the  company,  and  they  are 
interested  in  seeing  the  business  succeed.  Coupled  with 
loyalty  is  a  clear  chain  of  command  with  in-house  personnel. 
Because  people  on  the  premises  are  responsible  for  direct 
supervision  of  the  employees,  many  problems  are  minimized. 
Overtime  during  heavy  workload  periods  is  less  likely  to  be 
debated  and  response  time  is  better  when  quick  service  is 
required  (Carlyle, 1990;  Kelly, 1990). 

Further,  in-house  employees  tend  to  have  a  better  knowl¬ 
edge  of  the  organization's  business  and  to  know  the  users 
better.  They  do  not  feel  like  visitors  in  the  workplace  and 


are  more  likely  to  get  out  of  their  offices  and  explore 
other  work  areas  (Kelly, 1990;  Moad,1990). 

Finally,  several  authors  claim  the  organization  has 
better  control  over  the  quality  of  the  product  and  over 
disaster  recovery.  Information  system  personnel  are  part  of 
the  immediate  organizational  structure  and,  thus,  top  man¬ 
agement  has  oversight  responsibility  for  disaster  plans, 
quality  control  monitoring,  and  the  hiring  process  of  indi¬ 
viduals  (Kelly, 1990;  Wil liamson, 1991 ) . 

On  the  negative  side,  IS  employees  from  within  the 
organization  tend  to  fall  behind  their  outsourced  counter¬ 
parts  in  technical  expertise.  Costs  for  continuing  training 
can  amount  to  as  much  as  $10,000  to  $20,000  per  year  per 
person,  a  staggering  amount  in  the  minds  of  top  managers  who 
may  not  view  IS  training  as  a  fiscal  priority  (Kelly, 1990: 
106)  . 

Running  the  information  system  from  within  the  organiza¬ 
tion  often  costs  much  more  than  having  a  contracted  vendor 
do  it.  The  vendor  has  the  advantage  of  economies  of  scale, 
because  information  technology  is  its  entire  business  and  it 
may  hold  the  IS  contract  with  several  corporations.  Kelly 
cites  a  figure  of  50  million  instructions  per  second  (MIPS) 
as  the  lower  bound  for  economical  in-house  management. 

Below  that  level  of  information  load,  "the  existing  IS  staff 
will  be  hard  pressed  to  match  the  expense  ratios  available 
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to  an  outsourcing  vendor  that  can  spread  the  £ixed  costs 
over  a  higher  volume"  (Kel ly, 1990 : 103-104) . 

Outsourcing .  In  addition  to  the  lower  costs  and  in¬ 
creased  ability  to  adequately  train  personnel,  outsourced 
vendors  will  often  employ  more  technically  motivated  and 
competent  people.  One  of  the  reasons  behind  this  practice 
is  that  vendor  employees  can  more  realistically  "aspire  to  a 
job  in  senior  management"  than  they  can  in  an  organization 
that  is  not  in  business  solely  to  produce  information 
(Wil liamson, 1991 : 24) .  Many  IS  specialists  will  begin  in 
another  type  of  corporation  and  later  move  to  an  information 
company  because  of  the  greater  advancement  potential 
(Wil liamson, 1991) . 

As  was  discussed  earlier,  lack  of  loyalty,  ignorance  of 
the  corporate  business,  and  inability  to  interact  with  end- 
users  are  the  greatest  disadvantages  for  outsourcing 
(Wil liamson, 1991) . 

In  addition,  many  corporations  who  have  outsourced 
their  IS  operations  have  found  that  it  is  still  necessary  to 
keep  knowledgeable  systems  people  in  their  employ  "both  to 
manage  the  outsourcing  vendor  and  also  to  manage  the  needs 
of  users  within  the  firm"  (Kel ly , 1990 : 104) .  Thus,  neither 
the  IS  personnel  budget  nor  the  IS  training  budget  will  fall 
entirely  on  the  shoulders  of  the  vendor  (Kel ly, 1990 ) . 

Pinal ly,  at  least  two  authors  have  stated  that  once  the 
decision  is  made  to  outsource,  that  decision  is  probably 
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non-rescindabl e .  Reversing  the  plan  once  it  has  been  imple¬ 
mented  would  mean  building  and  training  a  competent  staff 
from  the  ground  up  (Kelly,1990;  Wi 1 1 iamson , 1991 ) . 

Outsourcing  information  services  is  becoming  increas¬ 
ingly  more  prevalent  in  corporate  America.  In  1990,  Hard 
says  it  was  a  $5  billion  industry  and  he  predicts  it  will 
grow  to  $50  billion  by  1994  (Hard, 1991 : 54) . 

Certainly,  any  organization  that  processes  information 
as  part  of  its  day-to-day  operation  should  consider  the 
applicability  of  outsourcing  its  IS  staff,  and  every  CIO 
should  keep  the  idea  at  the  forefront  at  all  times  even  if 
he  elects  not  to  pursue  it  (Carlyle, 1990 : 31) .  "Hospitals 
are  perfect  candidates  for  outsourcing  IS  because  they  are 
information-intensive,  yet  information  systems  are  not  their 
prime  business  function"  (Hard, 1991 : 54) .  According  to 
Kelly,  "It's  difficult  to  turn  back  from  outsourcing.  But 
if  the  decision  to  outsource  is  irrevocable,  the  decision 
not  to  is  infinitely  reviewable"  (Kel ly, 1990 : 106) . 

The  Air  Force  Medical  Service  is  keeping  the  idea  of 
outsourcing  hospital  information  systems  at  the  forefront 
with  its  various  methods  of  staffing,  (i.e.,  contracting 
with  civilians  as  at  Wilford  Hall  Medical  Center) 

(Obuchowski , 1992 )  and  using  communications  personnel  to 
perform  all  information  system  functions  as  in  Program 
Action  Directive  90-4  guidelines  (PAD  90-4,1990; 

Smal tz , 1991 ) .  The  advantages  and  disadvantages  of  each 
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source  and  the  best  staffing  method  in  serving  the 
customers'  needs  have  been  widely  argued  in  the  literature. 
The  researchers  set  out  to  determine  if  there  is  a  relation¬ 


ship  between  staffing  method  and  information  system  quality 
in  Air  Force  medical  centers. 

Standards  and  Goals  of  USAF  Medical  Information  Systems 
In  order  to  establish  the  first  level  (the  neutral 
level)  of  quality  in  medical  information  systems,  a  set  of 
standards  has  been  written  by  experts  at  the  policy  making 
level  in  the  Air  Force.  These  standards  are  found  in  Air 
Force  Regulation  168-4,  Chapter  14,  Medical  Information 
Systems  Management.  Air  Force  IS  management  objectives  are 
listed  below. 


USAF  MEDICAL  INFORMATION  SYSTEMS 
MANAGEMENT  OBJECTIVES 

1.  To  help  health  care  managers  at  all  levels  obtain 
timely  and  accurate  information  needed  to  plan, 
organize,  direct,  coordinate,  and  control  operations 
of  the  Medical  Service  according  to  the  priorities 

of  the  Air  Force  Surgeon  General. 

2.  To  establish  and  maintain  an  effective  IS  archi¬ 
tecture  to  support  the  Medical  Service  mission  con¬ 
sistent  with  the  Defense  Medical  Systems  Support 
Center  and  Air  Force  Plans. 

3.  To  simplify,  integrate,  and  modernize  health  care 
IS  policies  and  procedures  to  the  extent  possible 
consistent  with  a  coherent  approach  to  effective 
management  of  medical  information  resources. 

4.  To  ensure  physical,  administrative,  and  tech¬ 
nical  security  measures  and  management  standards 
are  adequate  for  protection  of  sensitive  unclassi¬ 
fied  health  care  and  patient  information,  Privacy 
Act,  and  classified  data. 


5.  To  establish  procedures  to  assess  the  effec¬ 
tiveness  and  operational  impacts  of  Medical  IS 
installations . 

6.  To  ensure  that  Medical  IS  applications  procured 
or  developed  are  compatible  with  existing  and 
planned  capabilities. 

7.  To  minimize  duplication  in  reporting,  data 
collection,  and  promoting  information  sharing 
among  medical  treatment  facilities  (MTF) ,  Major 
Command  Surgeon's  offices,  Headquarters  USAF 
Surgeon  General,  and  the  Defense  Medical  Systems 
Support  Center  (AFR  168-4,  Chap  14,  Para  14-3). 

In  addition,  the  regulation  spells  out  the  individual 
medical  center  CIO's  responsibilities  in  ensuring  his  or  her 
facility  meets  those  standards.  These  responsibilities  are 
found  in  Appendix  A.  Briefly,  Air  Force  medical  treatment 
facility  CIOs  are  responsible  for  determining  the  management 
and  clinical  information  needs  of  the  hospital  staff,  the 
level  of  detail  and  scope  of  the  needed  information,  the 
best  possible  method  of  providing  the  information,  and  the 
presentation  and  analysis  of  the  information.  Further,  they 
are  responsible  for  planning  for  future  information  needs 
and  system  requirements ;  maintenance,  monitoring,  and  secu¬ 
rity  of  all  information  systems  and  software;  and  the 
establishment  and  management  of  an  in-house  users'  group. 

The  specific  approach  taken  to  fulfill  CIO  responsibilities 
must  be  tailored  to  meet  the  needs  and  resources  of  the 
individual  facilities  (AFR  168-4,1990). 

Medical  information  systems  policy  is  initially 
developed  at  the  Department  of  Defense  level  by  an  agency 
entitled  Defense  Medical  Systems  Support  Center  (DMSSC) . 


42 


Those  policies  are  amended  and  tailored  to  meet  the  needs  of 
the  Air  Force  by  Medical  Service  Information  Systems 
Division  of  AFMSA  after  coordination  with  the  Air  Force 
Standard  Systems  Center  and  the  Air  Force  Surgeon  General's 
Office.  Finally,  the  Major  Command  Medical  Information 
Systems  Offices  further  adapt  the  guidance  to  fit  the  com¬ 
mand  mission  before  passing  it  down  to  the  MTF  CIOs  for 
implementation.  Figure  4  is  an  illustration  of  the  Air 
Force  Medical  Service's  medical  information  system  policy 
and  guidance  hierarchy. 

SYSTEM  POLICY  AND  GUIDANCE 


Figure  4.  Medical  System  Policy  and  Guidance  Hierarchy 
(AFR  168-4, 1990:Chap  14,  Para  14-2) 


Responsibility  for  reporting  to  the  Air  Force  Surgeon 
General's  Office  as  to  compliance  with  policy  at  each  MTF 
lies  with  the  Healthcare  Services  Management  Inspection 
(HSMI)  team.  This  team  inspects  every  Air  Force  MTF  every 
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12  to  24  months  using  checklists  similar  to  the  one  found  at 
Appendix  B  (AFR  168-4,1990).  During  their  last  inspections, 
all  six  Air  Force  medical  centers  were  found  to  be  satisfac¬ 
torily  complying  with  the  objectives  and  standards 
(Obuchowski ,1992 ) . 

This  section  served  to  answer  Investigative  Question  3: 
What  are  the  performance  standards  and  goals  of  the  USAF 
Medical  Service  as  set  by  the  Air  Force  Medical  Systems 
Agency  concerning  information  systems  at  USAF  medical  cen¬ 
ters?  The  explicit  methods  undertaken  by  the  six  sites  to 
meet  these  goals  will  be  discussed  in  the  next  chapter. 

In  addition,  this  section  served  to  partially  answer 
Investigative  Question  4:  To  what  extent  are  the  Medical 
Systems  Offices  meeting  the  standards  and  goals  set  by  the 
Air  Force  for  the  operation  of  medical  information  systems? 
All  six  research  sites  are  adequately  meeting  the  standards 
and  goals  of  the  USAF  concerning  information  systems.  In  so 
doing,  they  have  attained  at  least  a  neutral  level  of 
quality  by  conforming  with  standards.  Again,  further 
discussion  of  the  individual  programs  within  each  MSO  is 
warranted  and  is  found  in  the  next  chapter.  Establishing 
the  level  of  quality  above  the  neutral  point  must  be  done  by 
measuring  the  level  of  customer  satisfaction  as  illustrated 
earlier  in  this  chapter  in  Figure  2  (Ernst  and  Young, 1990). 

A  survey  of  information  system  customers  was  used  for  this 
measurement  and  will  be  described  in  the  next  chapter. 
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Summary  and  Conclusions 

This  chapter  established  the  need  for  quality  in  Air 
Force  Medical  Center  information  systems  by  examining  the 
changes  in  the  economic,  socio-political,  and  technologic 
environments  of  the  medical  care  industry.  The  importance 
of  quality  information  production  in  keeping  up  with  those 
changes  was  discussed. 

Furthermore,  measurement  criteria,  as  established  by 
experts,  for  gauging  the  quality  of  hospital  information 
systems  were  defined.  Specifically,  the  criteria  are: 
performance,  features,  reliability,  conformance,  durability, 
serviceability,  aesthetics,  and  perceived  quality.  These 
criteria  will  be  more  thoroughly  discussed  as  they  apply  to 
this  research  project.  This  section  served  to  answer  Inves¬ 
tigative  Question  2:  What  are  the  quality  indicators  neces¬ 
sary  to  measure  the  quality  of  information  systems  at  USAF 
medical  centers? 

The  researchers  discussed  technical  competence,  inter¬ 
personal  skills,  medical  industry  knowledge,  and  loyalty  to 
the  organization  as  requirements  of  information  system  staff 
members.  These  attributes  will  serve  as  a  topic  of  discus¬ 
sion  in  the  final  chapters  following  the  analysis  of  the 
findings  pertaining  to  staffing  methods  at  U.S.  Air  Force 
Medical  Center  information  system  offices. 

Lastly,  the  standards  and  goals  of  information  systems 
in  USAF  medical  facilities  were  discussed  as  outlined  in  Air 
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Force  Regulation  168-4.  These  standards  are  the  yardstick 
by  which  information  system  conformance  is  measured. 

Because  all  six  USAF  medical  centers  met  these  standards 
during  their  last  inspection  by  medical  information  system 
experts,  a  neutral  level  of  quality  was  established.  Thus, 
those  criteria  pertaining  to  customer  satisfaction  remained 
the  deciding  factors  for  the  establishment  of  quality  at  the 
six  research  sites.  This  section  provided  the  answer  to 
Investigative  Question  3:  What  are  the  performance  stand¬ 
ards  and  goals  of  the  USAF  Medical  Service  as  set  by  the  Air 
Force  Medical  Systems  Agency  concerning  information  systems 
at  USAF  medical  centers?  Further,  it  partially  answered 
Investigative  Question  4:  To  what  extent  are  the  Medical 
Systems  Offices  meeting  the  standards  and  goals  set  by  the 
Air  Force  for  the  operation  of  medical  information  systems? 
Further  discussion  of  Investigative  Question  4  will  be 
provided  in  the  next  chapter  in  a  summarization  of  how  each 
medical  center  information  system  office  monitors  the  status 
of  the  Air  Force's  standards  and  goals  within  their  individ¬ 
ual  facilities. 

The  following  chapter  will  address  the  methodology  used 
by  the  researchers  to  acquire  the  necessary  data  for  evalu¬ 
ating  the  level  of  quality  of  information  systems  at  USAF 
Medical  Centers  and  the  relationship  to  staffing  sources. 
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This  chapter  discusses  the  methodology  to  be  used  to 
answer  the  investigative  questions  that  were  posed  in  chap¬ 
ter  one.  Included  in  this  chapter  is  a  description  of  the 
four  research  phases;  a  description  of  the  measurement 
instruments  to  include  justification  of  their  use,  develop¬ 
mental  considerations,  and  the  verification  process;  a 
description  of  the  populations  under  study;  and  a  descrip¬ 
tion  of  the  analytical  tests  to  facilitate  interpretation  of 
the  data. 

Phases 


TABLE  1 

INVESTIGATIVE  QUESTION  RELATIONSHIPS  TO  RESEARCH  PHASES 


Phase  # 

Phase  Description 

Investiaative  Ouestion 

One 

Literature  Review 

2,3,4 

Two 

CIO  Information 
Surveys 

1,4 

Three 

Systems  Expert 
Interviews 

4 

Pour 

User  Surveys 

5 

The  research  was  completed  in  four  phases,  as  shown  in 
Table  1.  Phase  One  was  a  literature  review  to  gather  cur¬ 
rent  information  on  hospital  information  systems  and 
criteria  for  measuring  quality.  Phase  Two  was  the 
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development  and  administration  of  a  survey  of  USAF  Medical 
Center  CIOs,  the  purpose  of  which  was  to  gather  factual 
information  concerning  the  MSO  staff  and  functions  and 
available  information  systems.  In  the  third  phase,  informa¬ 
tion  was  gathered  about  the  major  information  systems  iden¬ 
tified  in  Phase  Two  in  order  to  more  objectively  evaluate 
the  computing  services  and  resources  available  at  each 
medical  center.  Finally,  Phase  Four  involved  the  develop¬ 
ment  and  administration  of  a  questionnaire  to  system  users 
at  each  medical  center.  Its  purpose  was  to  survey  opinions 
about  the  quality  of  information  services  provided  by  the 
MSOs . 

Phase  One:  The  Literature  Review.  Phase  one  involved  a 
review  of  the  existing  literature  on  trends  in  hospital 
information  systems  and  measuring  quality  in  these  systems, 
and  personal  interviews  with  officials  at  the  policy  making 
level  at  the  Air  Force  Medical  Systems  Agency  (AFMSA) .  This 
initial  phase  was  necessary  to  answer  the  first  and  second 
investigative  questions  and  to  partially  answer  the  fourth 
investigative  question: 

Question  2.  What  are  the  quality  indicators  necessary 
to  measure  the  quality  of  information  systems  at  USAF  medi¬ 
cal  centers? 

Question  3.  What  are  the  performance  standards  and 
goals  of  the  USAF  Medical  Service  concerning  information 
systems  at  USAF  medical  centers? 
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Question  4.  To  what  extent  are  the  Medical  Systems 
Offices  meeting  the  standards  and  goals  set  by  the  Air  Force 
Medical  Systems  Agency  concerning  information  systems  at 
USAF  medical  centers? 

The  review  of  the  literature  was  conducted  using  an 
on-line  search  of  the  Defense  Technical  Information  Center 
(DTIC)  and  review  of  the  holdings  at  the  Air  Force  Institute 
of  Technology  (AFIT) ,  Wright-Patterson  Medical  Center,  and 
Wright  State  University  medical  and  general  libraries.  The 
findings  of  the  literature  review  were  presented  in  Chapter 
II. 

Three  goals  were  achieved  in  the  literature  review 
phase.  First,  criteria  were  documented  for  highly  qualified 
information  system  specialists  in  order  for  hospitals  to 
keep  pace  with  the  rapidly  advancing  world  of  hospital 
information  systems.  The  various  types  of  qualified  infor¬ 
mation  system  specialists  (i.e.,  people  directly  employed  by 
the  organization  and  contractors)  were  discussed.  Informa¬ 
tion  concerning  the  types  used  by  USAF  MSOs  was  obtained  in 
Phase  Two  of  the  research. 

The  second  goal,  to  define  and  establish  criteria  for 
quality  in  Air  Force  medical  center  information  systems,  was 
partially  accomplished  by  searching  the  literature  on  qual¬ 
ity.  Garvin's  "eight  dimensions  of  quality"  provided  a 
framework  for  measuring  a  product's  effectiveness  in  ful¬ 
filling  the  desires  of  customers.  These  eight  dimensions 
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are:  performance,  features,  reliability,  conformance, 
durability,  serviceability,  aesthetics,  and  perceived  qual¬ 
ity  (Garvin, 1988) .  The  researchers  feel  that  durability, 
usually  measured  by  the  length  of  the  product's  life,  is 
outside  the  scope  of  measuring  the  quality  of  information 
systems.  This  conclusion  was  reached  because  the  ultimate 
product  of  an  information  system  is  the  information  it 
provides.  The  hardware  and  software  that  produce  the  infor¬ 
mation  will  periodically  be  replaced  and  updated,  but  the 
lifespan  of  the  information  itself  is  dictated  solely  by 
external  factors  (Ahituv  and  Neumann, 1990 ) .  However,  the 
remaining  seven  dimensions  were  used  as  specific  criteria 
for  measuring  the  quality  of  information  systems  at  the 
medical  centers.  Table  2  shows  the  phase  in  which  each 
quality  dimension  was  measured.  Those  phases  and  instru¬ 
ments  not  yet  discussed  will  be  covered  in  following  sec¬ 
tions  . 


TABLE  2 


QUALITY  DIMENSIONS  AND  PHASES  OP  MEASUREMENT 


Dimension 

Phase 

Instrument 

Performance 

One 

Literature  Review 

Two 

CIO  Survey 

Three 

System  Summary 

Four 

User  Survey 

Features 

Three 

System  Summary 

Four 

User  Survey 

Reliability 

Three 

System  Summary 

50 


TABLE  2  (continued) 


Conformance 

One 

Literature  Review 

Two 

CIO  Survey 

Three 

System  Summary 

Serviceability 

Two 

CIO  Survey 

Three 

System  Summary 

Four 

User  Survey 

Aesthetics 

Four 

User  Survey 

Perceived  Quality 

Four 

User  Survey 

The  third  goal  of 

the  literature 

research  phase  was 

establish  guidelines  for  evaluating  performance  and  conform¬ 
ance  of  the  services  provided  by  USAF  Medical  Center  MSOs. 
The  intent  of  the  researchers  was  to  determine  what  perform¬ 
ance  standards  and  goals  exist  for  Air  Force  medical  infor¬ 
mation  systems  and  the  organizational  level  at  which  those 
standards  and  goals  are  formulated.  Air  Force  Regulation 
168-4,  Administration  of  Air  Force  Medical  Facilities, 
provided  the  researchers  with  insight  into  computer  system 
development  and  monitoring  procedures  utilized  by  the  USAF 
medical  service  (1990).  The  specific  objectives  of  USAF 
medical  information  systems  management  are  listed  in  Chapter 
II.  Further,  the  specific  responsibilities  of  the  medical 
facility  CIO  for  meeting  those  objectives  are  found  in 
Appendix  A. 

The  information  gathered  from  published  literature 
provided  the  researchers  with  a  framework  for  developing  the 
criteria  for  measuring  quality  in  medical  information  sys¬ 
tems  and  the  standards  and  goals  of  USAF  medical  information 
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systems,  thereby  answering  Investigative  Questions  2,  3,  and 
4. 

The  next  logical  step  in  the  research  was  to  use  the 
quality  criteria  and  standards  established  in  this  phase  to 
evaluate  the  services  provided  by  each  MSO.  This  process  is 
discussed  in  the  next  section. 

Phase  Two:  The  CIO  Survey.  Phase  Two  involved  the 
development  and  administration  of  the  first  of  two  survey 
instruments .  The  purpose  of  this  survey  was  to  answer 
Investigative  Questions  1  and  4: 

Question  1 .  What  is  the  source  of  each  Medical  Systems 
Office  staff  member  at  each  medical  center  (i.e.,  medical 
personnel  with  additional  training,  communications  squadron 
personnel,  or  contractor)? 

Question  4.  To  what  extent  are  the  Medical  Systems 
Offices  meeting  the  standards  and  goals  set  by  the  Air  Force 
for  the  operation  of  medical  information  systems? 

In  order  to  fully  answer  these  investigative  questions, 
specific,  factual  information  was  needed  from  the  CIOs  at 
each  medical  center.  The  researchers  designed  a  question¬ 
naire  for  this  purpose. 

Population.  The  population  of  interest  for  this 
phase  of  the  research  are  the  chief  information  officers  at 
each  of  the  six  continental  United  States  (CONUS)  medical 
centers  in  the  Air  Force. 
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Air  Force  hospitals  are  divided  into  three  categories: 
hospitals,  regional  hospitals,  and  medical  centers.  Medical 
centers  are  distinguished  from  the  other  categories  because 
they  have  a  higher  number  of  inpatient  beds,  provide  a  wider 
range  of  medical  services,  receive  referrals  from  the  lower 
level  hospitals,  and  provide  specialty  training  to  medical 
professionals  (AFR  168-4,1990).  The  six  USAF  medical  cen¬ 
ters  in  CONUS  are  located  at  Andrews  AFB ,  Maryland,  Keesler 
AFB,  Mississippi,  Lackland  AFB,  Texas,  Scott  AFB,  Illinois, 
Travis  AFB,  California,  and  Wright-Patterson  AFB,  Ohio. 

As  stated  in  Chapter  I,  this  study  was  limited  to  the 
medical  centers  because  the  researchers  were  interested  in 
keeping  the  population  fairly  homogeneous  so  reasonable 
comparisons  were  possible.  The  medical  centers  have  more 
highly  developed  information  systems  with  large  full-time 
staffs  due  to  the  nature  of  the  medical  mission.  Many  of 
the  smaller  hospitals  are  limited  in  their  automated  infor¬ 
mation  services  because  they  have  part-time  CIOs  and  because 
they  have  fewer  monetary  and  labor  resources  from  which  to 
draw  (Constantian,1991) . 

During  Phase  Two,  the  survey  respondents  were  the  CIOs 
at  each  medical  center.  The  objective  of  this  phase  was  to 
obtain  factual  data  pertaining  to  the  demographics  of  the 
MSO  staff,  the  available  information  systems,  and  monitoring 
and  control  procedures.  The  researchers  felt  that  the  CIO 
at  each  medical  center  was  the  best  source  for  factual. 
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statistical  data  about  operations  and  staffing  in  his  MSO. 

In  addition,  with  the  strong  emphasis  on  quality  and  plan¬ 
ning  throughout  the  Air  Force  Medical  service,  the  research¬ 
ers  felt  that  the  information  provided  by  the  CIOs  could 
possibly  be  supplemented  by  a  detailed,  accurate  mission 
statement.  The  researchers  surveyed  every  USAF  Medical 
Center  CIO  to  obtain  a  total  census  of  the  population. 

Measurement  Instrument.  Written  surveys  were 
determined  to  be  the  appropriate  method  for  collecting  the 
information  required  in  Phase  Two  for  several  reasons. 

First,  there  was  a  large  geographical  distance  between  the 
researchers  and  most  of  the  survey  population  making  person¬ 
al  interviews  infeasible  at  all  but  one  location,  Wright  - 
Patterson  AFB.  Telephone  interviews  were  ruled  out  because 
the  researchers  felt  that  the  survey  participants  would 
require  some  time  to  gather  the  data  once  they  were  informed 
what  was  needed.  Moreover,  a  written  survey  ensured  that 
all  CIOs  were  asked  the  same  questions  in  exactly  the  same 
way  and  that  transcription  errors  by  the  researchers  would 
not  occur,  common  problems  with  telephone  interviews.  The 
CIO  survey  is  found  at  Appendix  C. 

Each  of  the  questions  was  followed  by  a  large  blank 
area  in  which  the  respondents  could  write  as  much  additional 
information  as  they  felt  necessary  for  the  researchers  to 
obtain  an  accurate  picture  of  the  nature  of  each  MSO's 
staffing  source,  work  load,  and  functions.  The  researchers 
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remained  available  via  telephone  to  answer  questions  from 
the  CIOs  pertaining  to  the  intent  of  the  survey.  During 
initial  telephone  conversations,  the  researchers  found  a 
great  deal  of  interest  in  the  results  of  this  research  among 
the  CIOs.  Therefore,  adequate  and  detailed  information  was 
expected  from  all  six  of  the  CIOs.  Additionally,  the 
researchers  followed  up  by  telephone  if  more  information  was 
required  from  the  respondents.  This  method  of  combining 
written  questionnaires,  in  order  for  the  CIOs  to  have  ade¬ 
quate  time  to  retrieve  required  statistical  data,  with 
follow-up  telephone  conversations  proved  to  be  effective  in 
obtaining  consistent  information  among  the  medical  centers. 

Development  of  the  Instrument.  The  questions 
included  in  this  survey  were  developed  by  the  researchers 
based  on  knowledge  gained  from  published  literature,  prelim¬ 
inary  discussions  with  the  CIO  at  Wright-Patterson  Medical 
Center,  and  discussions  with  the  Medical  Service  Information 
Systems  Division  Chief  at  AFMSA .  Individual  questions  were 
designed  with  the  objective  of  finding  answers  to  Investiga¬ 
tive  Questions  1  and  4  at  the  forefront.  In  writing  the 
questions,  the  researchers  followed  the  guidance  of  Emory 
and  Cooper  (1991)  concerning  question  wording  to  guard 
against  bias  in  the  measurement  questions. 

The  questions  in  the  CIO  survey  were  open-ended  for 
several  reasons.  First,  this  is  the  most  appropriate  style 
of  question  when  the  objective  is  to  obtain  sources  of 
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information  and  factual  data  (Emory  and  Cooper, 1991) . 

Second,  the  range  of  possible  responses  were  too  numerous  to 
list  in  the  closed  question  structure.  Further,  the  range 
of  possible  responses  could  not  be  predicted  by  the 
researchers.  Therefore,  multiple-choice  questions  with 
exhaustive  sets  of  possible  answers  were  infeasible.  Multi¬ 
ple-choice  questions  without  every  possible  answer  provided 
as  a  choice  are  inappropriate  in  scientific  research  (Emory 
and  Cooper, 1991) .  Finally,  all  six  CIOs  seemed  motivated  to 
provide  detailed  answers  to  the  survey  questions  during 
initial  telephone  conversations  with  the  researchers.  Open- 
ended  questions  are  well  suited  to  a  willing  and  small 
survey  population  (Emory  and  Cooper , 1991 ) . 

The  purpose  of  the  first  section  of  the  survey  was  to 
establish  reliable  points  of  contact  at  each  medical  center. 
In  so  doing,  the  researchers  were  able  to  follow  up  on  the 
survey's  progress  fairly  easily.  Included  were  questions 
pertaining  to  Defense  Data  Network  electronic  mail  addresses 
and  Defense  Systems  Network  telephone  number,  along  with  the 
individual's  name  who  would  be  distributing  the  Phase  Four 
user  surveys.  This  information  provided  the  researchers 
with  several  avenues  for  communications  with  the  respond¬ 
ents  . 

The  next  section  of  the  CIO  survey  was  designed  to 
provide  the  answer  to  Investigative  Question  1.  It  soli¬ 
cited  demographic  information  about  each  medical  center's 
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information  system  office  staff.  Respondents  were  asked  how 
many  people  were  on  their  staffs  and  to  categorize  the 
staff  members  according  to  Air  Force  Specialty  Code  (AFSC), 
civilian  occupational  code  title,  or  externally  contracted 
information  service  provider.  Civilian  occupational  code 
titles  and  AFSCs  are  an  indication  of  the  nature  and  level 
of  occupational  training  the  individual  has  received 
(AFR  39-1,1991).  The  CIOs  were  further  asked  whether  the 
staff  members  were  supervised  directly  by  the  medical  cen¬ 
ter,  by  the  Air  Force  communications/ computer  community,  or 
by  a  contractor.  The  answers  to  these  questions  enabled  the 
researchers  to  group  staff  members  into  the  following  cate¬ 
gories:  medical  personnel  with  additional  computer  train¬ 

ing,  communications  squadron  personnel,  or  contractor  per¬ 
sonnel  .  The  communications  squadron  personnel  were  further 
grouped  according  to  whether  they  are  directly  supervised  by 
individuals  within  the  medical  center  or  by  individuals 
within  the  communications/computer  community  outside  the 
medical  center. 

In  the  third  section,  the  CIOs  were  asked  to  list  the 
information  systems  in  service  at  their  medical  centers  and 
the  types  of  system  control  statistics  that  are  regularly 
monitored . 

The  system  control  statistic  information  was  used  to 
further  answer  Investigative  Question  4.  Chapter  II  reports 
the  findings  that  all  six  medical  centers  satisfactorily 
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meet  the  standards  and  goals  of  the  Air  Force  in  providing 
information  services  to  the  hospital  staff.  The  researchers 
were  interested  in  finding  out  how  each  MSO  judges  its  own 
level  of  success;  by  measuring  the  amount  of  system  down 
time,  time  between  failures,  time  to  repair,  or  any  of  a 
number  of  other  measures  of  system  quality  (Garvin, 1988 ) . 

The  information  concerning  available  systems,  budgets, 
and  MSO  staff  to  medical  center  staff  ratio  were  included  to 
provide  additional  insight  into  the  individual  MSO's  ability 
to  provide  additional  services  to  its  customers  and  the 
constraints  under  which  the  information  specialists  are 
working These  constraints  will  be  one  of  the  topics  of 
discussion  in  Chapter  V. 

Pretesting.  The  draft  CIO  questionnaire  underwent 
critical  evaluation  prior  to  its  distribution  to  the  survey 
population.  First,  it  was  reviewed  by  the  thesis  advisors 
for  completeness  and  appropriate  wording.  Following  their 
review,  the  survey  was  given  to  a  medical  systems  expert  in 
the  Command  Surgeon's  office  at  Air  Force  Materiel  Command 
Headquarters  (HQ  AFMC)  and  two  AFIT  graduate  students  who 
are  U.S.  Army  Medical  Service  Corps  officers.  The  recommen¬ 
dations  of  these  experts  resulted  in  several  minor  revisions 
to  the  questionnaire.  Given  the  factual  nature  of  the 
questions  on  the  CIO  survey  and  the  fact  that  the  survey 
participants  were  identified  and  easily  accessible  for 
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follow-up  for  answer  clarification,  further  pretesting  of 
this  instrument  was  deemed  unnecessary  by  the  researchers. 

Finally,  the  CIO  survey  was  passed,  along  with  the 
Phase  Four  survey  instrument,  through  official  channels  to 
the  Medical  Systems  Directorate  of  the  Air  Force  Office  of 
the  Surgeon  General  at  AFMSA  for  final  approval .  This 
approval  was  granted  on  1  May  1992,  with  required  changes 
only  to  the  Phase  Four  survey.  This  second  survey  will  be 
discussed  in  detail  in  a  later  section  of  this  chapter. 

Data  Collection  Plan.  The  CIO  surveys  were  dis¬ 
tributed  to  the  medical  center  CIOs  via  data  facsimile  on 
22  May  1992.  Each  of  the  officers  was  given  a  thorough 
explanation  of  the  survey  over  the  telephone  prior  to  its 
being  sent  and  further  questions  were  answered  by  the 
researchers  as  the  respondents  were  going  though  the  process 
of  completing  the  surveys.  All  six  surveys  were  sent  back 
to  the  researchers  by  14  July  1992. 

Survey  Follow-Up  Procedures.  The  researchers 
telephoned  the  six  recipients  of  the  CIO  survey  just  prior 
to  and  immediately  after  the  questionnaire  was  sent  to  them. 
In  addition,  three  to  four  follow-up  telephone  calls  from 
the  researchers  to  the  CIOs  to  check  on  survey  completion 
progress  were  the  norm.  Five  of  the  six  surveys  were 
returned  to  the  researchers  within  two  weeks  after  they  were 
sent.  One  survey  was  somewhat  delayed  in  its  return.  At 
that  location,  there  was  a  misunderstanding  concerning 
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survey  procedure.  The  sixth  completed  survey  was  returned 
to  the  researchers  within  a  short  time  after  the  issue  was 
resolved  for  a  survey  response  rate  of  100  percent. 

Statistical  Tests.  Results  from  the  information 
systems  officer  surveys  were  summarized  by  the  researchers. 
These  summaries  are  in  Chapter  IV  and  will  be  discussed  in 
detail  in  that  chapter.  The  MSO  data  were  not  amenable  to 
statistical  analysis  other  than  by  categorical  summary  of 
the  individual  MSOs  according  to  staffing  source.  Instead, 
it  was  used  as  a  basis  for  comparison  of  the  user  survey 
results  among  the  six  medical  centers. 

Phase  Three:  Air  Force  Medical  Systems  Data.  Part  of 
the  data  acquired  from  the  individual  medical  center  CIOs 
involved  a  listing  of  information  systems  in  service  at  each 
of  the  six  locations.  There  was  little  uniformity  among  the 
medical  centers  in  the  types  of  systems  used  to  provide 
information  support.  Further,  many  of  these  systems  have 
been  in  service  for  a  number  of  years  while  others  are  very 
recent  additions. 

The  researchers  and  the  research  sponsor  determined 
that  additional  information  was  needed  describing  the  major 
medical  information  systems  to  include  maintenance  require¬ 
ments,  manpower  requirements,  and  quantity  and  quality  of 
information  provided  by  each  system  (Obuchowski , 1992 ) .  This 
information  gave  the  researchers  further  insight  into  the 
interpretation  and  analysis  of  the  Phase  Four  user  survey 
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instrument.  Detailed  descriptions  of  the  major  information 
systems  at  USAF  medical  centers  were  obtained  from  a  peri¬ 
odic  report  compiled  by  AFMSA  as  required  by  Air  Force 
Regulation  168-4,  Chapter  14. 

Phase  Four:  The  User  Survey.  As  discussed  in  the  first 
research  phase  section  reported  in  this  chapter,  four  of 
Garvin's  measures  of  quality  were  to  be  gauged  using  the 
subjective  opinions  of  information  system  customers 
(Garvin, 1988) .  Therefore,  the  aggregate  opinions  of  system 
users  at  each  medical  center  were  used  to  answer  Investiga¬ 
tive  Question  5:  To  what  extent  do  the  customers  feel  that 
the  MSO  is  providing  them  with  quality  technical  support  in 
information  systems?  With  this  objective  in  mind,  the 
researchers  designed  and  administered  a  survey  to  a  sample 
of  the  users  at  each  research  site. 

Measurement  Instrument.  As  in  Phase  Two,  the  mail 
survey  was  determined  by  the  researchers  to  be  the  most 
appropriate  instrument  for  collecting  the  information 
required  to  meet  the  objective  of  Phase  Four.  This  survey 
method  was  chosed  for  several  reasons.  First,  the  anonymity 
of  the  respondents  was  extremely  important  in  order  to 
receive  honest  answers  to  opinion  questions.  Specific 
answers  can  easily  be  attributed  to  specific  individuals  in 
personal  and  telephone  interviews  (Emory  and  Cooper , 1991 ) . 
Second,  the  researchers  felt  that  the  respondents  would  need 
extra  time  to  think  about  the  survey  questions  and  to  verify 
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answers.  The  sel f -administered  mail  questionnaire  allows 
the  respondent  to  take  as  much  time  as  they  need  to  answer 
the  questions  (Emory  and  Cooper , 1991 ) .  Therefore,  both 
personal  and  telephone  interviews  were  infeasible  and  unde¬ 
sirable. 

Moreover,  the  researchers  had  a  limited  amount  of  time 
in  which  to  conduct  their  study.  One  of  the  advantages  of 
mail  surveys  over  personal  or  telephone  interviews  is  that 
the  surveys  will  reach  respondents  who  might  otherwise  be 
inaccessible  (Emory  and  Cooper , 1991 ) . 

Emory  and  Cooper  list  several  drawbacks  to  mail  sur¬ 
veys.  The  researchers  overcame  these  drawbacks  in  the 
design  and  distribution  method  of  the  instrument.  First, 
mail  surveys  have  consistently  produced  lower  response  rates 
than  personal  or  telephone  interviews  (Emory  and  Cooper, 
1991).  Based  on  the  experiences  of  other  researchers  who 
have  surveyed  Air  Force  medical  personnel,  specifically 
those  of  Parker  (1987)  and  Constantian  (1990),  a  response 
rate  of  approximately  40%  was  anticipated. 

A  second  limitation  of  mail  surveys  lies  in  the  quanti¬ 
ty  and  kind  of  data  that  can  be  gathered.  Probing  and/or 
complex  questions  are  not  positively  received  by  mail  survey 
respondents  (Emory  and  Cooper , 1991 ) .  Therefore,  relatively 
simple  questions  were  written  with  short  multiple-choice 
answers.  Definitions  were  provided  for  terms  that  might  be 
confusing  to  the  respondents  (detailed  in  the  next  section). 
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In  addition,  the  survey  was  designed  to  take  a  short  time  to 
complete.  During  pretesting,  the  user  survey  was  timed  to 
take  an  average  of  14  minutes  to  complete. 

Development  of  the  Instrument.  During  Phase  Four 
of  the  research,  a  sel f -administered  survey  was  used  to 
collect  data  from  a  sample  of  the  information  system  user 
population  at  the  six  medical  centers.  This  survey  is  found 
at  Appendix  D.  The  researchers  surveyed  the  administrative 
and  clinical  customers  of  the  MSOs  to  determine  their  atti¬ 
tudes  and  preferences  about  the  quality  of  technical  support 
in  information  systems.  The  purpose  of  this  survey  was  to 
answer  Investigative  Questions  4  and  5.  The  researchers 
used  multiple-choice  questions  and  statements  with  a  Likert- 
type  scale.  According  to  Emory  and  Cooper  (1991),  multiple- 
choice  questions  are  appropriate  for  determining  gradations 
of  preference,  interest,  or  agreement  while  summated  scales 
are  appropriate  for  measuring  a  respondent's  attitude  toward 
the  object  of  interest. 

The  first  few  questions  asked  for  factual  background 
infoxmation  about  the  administrators  and  clinicians  followed 
by  qtestions  to  determine  their  attitudes  and  preferences 
related  to  the  types  and  quality  of  information  services 
offered  by  the  MSOs.  The  survey  began  with  questions  that 
were  straightforward  and  that  required  little  thought  on  the 
part  of  the  respondent.  As  the  respondent  progressed 
through  the  survey,  the  questions  became  increasingly 
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probing.  This  method,  recommended  by  Churchill,  gradually 
draws  in  the  interest  of  the  participant,  and  previous 
research  has  suggested  that  it  increases  the  likelihood  that 
the  survey  will  be  completed  by  the  respondent.  As  the 
questions  increase  in  depth,  they  encourage  the  participant 
to  reflect  upon  his  or  her  responses,  thus  providing  greater 
theoretical  validity  to  the  mere  difficult  questions 
(Churchill ,1991:231-232) . 

Part  one  of  the  questionnaire  specifically  sought  to 
define  personal  characteristics  of  the  user  such  as  job 
duties,  previous  computer  experience  and  training,  and  level 
of  computer  use  while  on  duty.  Great  care  was  taken  in 
formulating  the  demographic  questions  such  that  respondent 
anonymity  was  preserved.  This  data  was  requested  so  that 
the  researchers  could  check  for  and  evaluate  any  correlation 
between  the  user's  computer  knowledge  level  and  level  of 
satisfaction  with  the  medical  center's  information  systems. 

In  part  two,  the  user  was  asked  to  specify  the  three 
information  systems  they  use  most  often  while  performing 
their  duties  and  then  to  rate  those  three  systems  on  per¬ 
formance,  additional  features,  serviceability,  and  aesthet¬ 
ics.  These  evaluation  criteria  are  four  of  the  five  most 
appropriately  measured  by  the  customers  from  Garvin's  list 
of  quality  dimensions  as  discussed  earlier  in  this  chapter 
(Garvin, 1988) .  Giving  the  respondent  the  option  to  rate  up 
to  three  separate  information  systems  was  a  recommendation 
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from  the  sponsor  (Obuchowski , 1992 )  .  It  was  felt  that  the 
different  systems  in  service  at  each  medical  center  offer 
different  levels  of  services  and  are  at  different  stages  in 
their  development  life  cycles.  If  the  user  were  required  t 
rate  all  the  systems  he  or  she  uses  with  a  composite  score 
for  each  factor,  the  user  might  tend  to  skew  the  score 
toward  the  one  with  which  he  or  she  is  least  satisfied. 
Therefore,  the  four  satisfaction  scores  (performance,  addi¬ 
tional  features,  serviceability,  and  aesthetics)  would  not 
be  a  true  indication  of  the  user’s  level  of  satisfaction 
with  overall  medical  information  services  (Obuchowski : 1992 ) 
Moreover,  the  researchers  were  of  the  opinion  that  some  of 
the  factors  of  satisfaction  might  be  of  greater  importance 
to  the  users  than  others.  Thus,  the  users  were  given  the 
opportunity  to  rate  the  level  of  importance  they  place  on 
each  factor  in  each  system  they  rated.  These  importance 
ratings  were  used  by  the  researchers  to  weight  the  satisfac 
tion  scores.  The  method  of  weighting  will  be  detailed 
further  in  the  Statistical  Tests  section  of  this  chapter. 

Finally,  part  three  asked  the  user  to  rate  his  or  her 
overall  satisfaction  with  the  MSO  staff  and  the  medical 
center  information  systems.  The  objective  of  this  section 
was  to  measure  the  customers'  level  of  perceived  quality  of 
medical  information  services.  Perceived  quality  is  the 
fifth  of  Garvin's  dimensions  most  appropriately  judged  by 
the  system  users  (Garvin , 1988 ) . 


All  questions  that  surveyed  the  opinions  of  the 
respondents  were  based  on  a  5-point  Likert  scale.  The  scale 
of  responses  for  all  questions  requiring  a  subjective  answer 
were  considered  to  be  interval.  Although  the  respondents 
had  six  answers  from  which  to  choose,  answer  number  one  was 
"No  opinion."  and  was,  therefore,  considered  to  a  be  nonre¬ 
sponse  and  not  on  the  Likert  scale.  Respondent  demographic 
data  was  gathered  using  multiple-choice  answers  with  each 
choice  considered  a  discrete  value. 

The  questions  in  this  survey  were  designed  to  circum¬ 
vent  misunderstandings  by  avoiding  generalizations  or  assum¬ 
ing  previous  knowledge  of  technical  terminology.  For  exam¬ 
ple,  the  respondent  was  asked  to  rate  his  or  her  level  of 
satisfaction  with  and  how  important  he  or  she  considered 
several  factors  (i.e.,  performance,  additional  features, 
serviceability,  and  aesthetics)  in  medical  information 
systems.  Before  each  specific  question,  the  researchers 
provided  a  detailed  definition  of  the  factor. 

Pretesting.  The  draft  user  survey  was  critically 
evaluated  prior  to  its  distribution  to  the  users  to  ensure 
that  the  researchers'  objectives  in  survey  design  were 
achieved.  The  researchers  used  a  multiphased  approach  for 
pretesting  the  instruments. 

As  with  the  CIO  survey,  the  Phase  Four  measurement 
instrument  was  first  submitted  to  thesis  advisors  to  review 
for  completeness  and  appropriateness.  Next,  the  CIO  and  a 
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nursing-information  systems  liaison  officer  who  is  a  part- 
time  APIT  student,  both  stationed  at  Wright-Patterson 
Medical  Center,  were  called  upon  for  their  recommendations. 
Finally,  the  medical  systems  expert  at  HQ  APMC  reviewed  the 
survey. 

The  second  phase  of  pretesting  the  user  survey 
consisted  of  administering  the  questionnaire  to  fifteen 
graduate  students  at  APIT.  Twelve  of  those  taking  the 
pretest  were  students  in  Information  Resource  Management 
program  and  the  remaining  three  were  U.S.  Army  and  Air  Force 
Medical  Service  Corps  officers  in  other  programs.  All  were 
asked  to  use  the  computer  systems  and  the  information  system 
office  at  AFIT  as  the  basis  for  their  answers  to  the  survey 
questions.  The  fact  that  those  taking  the  pretest  were 
evaluating  educational  information  systems  rather  than 
medical  information  systems  was  not  relevant  to  the  objec¬ 
tives  of  pretesting  the  instrument. 

The  user's  survey  took  an  average  of  slightly  more  than 
14  minutes  for  the  pretest  group  (range  -  9.25  to  17.5 
minutes).  Both  the  pretest  group  and  the  systems  experts 
made  several  recommendations,  which  led  to  minor  modifica¬ 
tions  to  the  survey. 

Finally,  the  user  survey,  along  with  the  CIO  survey, 
was  passed  through  official  channels  to  the  Medical  Systems 
Directorate  of  the  Air  Force  Office  of  the  Surgeon  General 
(AFMSA)  for  final  approval .  This  approval  was  granted  on 
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1  May  1992,  with  the  only  required  change  being  the  addition 
of  the  respondent's  option  to  choose  which  information 
systems  he  or  she  would  evaluate  on  the  user  survey. 

Sampling  Plan.  In  Phase  Pour,  the  population  of 
interest  consisted  of  both  administrative  and  clinical 
personnel  who  are  the  users  of  the  information  systems  at 
each  of  the  six  medical  centers.  The  reasons  for  limiting 
this  study  to  only  the  medical  centers  are  discussed  under 
Phase  Two.  This  population  represents  the  customers  of  the 
MSO  at  each  medical  center. 

The  researchers'  goal  was  to  send  out  approximately  300 
surveys  to  each  site.  In  so  doing,  the  researchers  were 
confident  that  they  would  have  an  adequate  return  rate  to 
use  the  survey  data  to  make  assumptions  about  the  entire 
population  in  accordance  with  the  Law  of  Large  Numbers 
(McClave  and  Benson, 1991) .  The  survey  response  means  are  an 
unbiased  estimator  of  the  response  means  if  the  entire 
population  were  surveyed  independent  of  sample  size  accord¬ 
ing  to  the  Central  Limit  Theorem  (McClave  and  Benson , 1991 ) . 
From  the  CIO  survey,  the  researchers  learned  that  there  were 
approximately  12,500  information  system  users  at  the  medical 
centeis.  Therefore,  about  14  percent  of  users  were  sent 
surveys  in  order  to  average  300  surveys  at  each  medical 
center . 

The  user  surveys  were  packaged  in  bulk  and  mailed  to 
the  individual  points  of  contact  within  each  MSO  on  22  May 
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1992.  Each  point  of  contact  was  given  distribution  instruc¬ 
tions  as  shown  in  Appendix  E.  The  researchers'  plan  was  to 
have  the  points  of  contact  use  their  Terminal  Area  Security 
Officers  (TASO)  as  sub-distribution  points. 

As  explained  earlier,  the  researchers'  goal  was  to  give 
surveys  to  approximately  14%  of  the  systems  users  at  each 
medical  center  and  to  send  an  average  of  about  300  surveys 
to  each  site.  This  number  would  ensure  that  at  least  30 
surveys  would  be  returned  from  each  site  given  a  pessimistic 
response  rate  of  10  to  15  percent.  The  total  number  of 
users  at  each  site  was  taken  from  the  CIO  survey.  At  all 
six  medical  centers,  the  total  number  of  users  equaled  the 
total  number  on  the  medical  center  staff.  As  relayed  to  the 
researchers  during  telephone  conversations,  all  six  CIOs 
considered  everyone  employed  by  their  medical  centers  to  use 
medical  information  systems  either  directly  or  indirectly. 
Therefore,  all  medical  center  staff  members  are  MSO  custom¬ 
ers  . 

Initial  survey  distribution  numbers  were  calculated  by 
taking  14  percent  of  the  total  number  on  staff  at  each 
medical  center.  Four  of  the  six  distribution  numbers 
(Andrews  AFB,  Keesler  AFB,  Travis  AFB,  and  Wright-Patterson 
AFB)  were  considered  to  be  adequate  to  produce  at  least  30 
responses  from  each  site.  However,  Scott  AFB  and  Lackland 
AFB 's  numbers  were  adjusted. 
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Fourteen  percent  of  Scott's  1000  users  is  140.  The 
researchers  felt  that  if  only  140  of  Scott's  users  were  sent 
surveys,  fewer  than  30  might  be  returned.  Therefore,  the 
distribution  number  was  adjusted  to  230,  the  lowest  number 
among  the  other  sites. 

The  medical  center  at  Lackland  AFB  has  4500  staff 
members,  and  its  initial  distribution  number  was  calculated 
to  be  630.  The  researchers  adjusted  this  number  downward  to 
450  out  of  consideration  for  the  point  of  contact  in  the 
MSO .  They  felt  that  the  distribution  of  630  surveys  would 
place  an  inordinately  large  additional  workload  on  one 
person  and  might  decrease  her  level  of  cooperation  with  and 
enthusiasm  for  the  research  project. 

The  distribution  rate  at  each  medical  center  was  calcu¬ 
lated  by  dividing  the  distribution  number  into  the  total 
number  on  staff.  With  the  exceptions  of  Scott  and  Lackland, 
one  survey  needed  to  be  distributed  per  seven  users. 

Because  of  the  distribution  number  adjustments  described 
earlier,  Scott's  distribution  rate  was  one  survey  per  five 
users,  and  Lackland's  distribution  rate  was  one  survey  per 
ten  users. 

Table  3  shows  distribution  numbers  and  distribution 
rates  for  the  medical  centers. 

Every  TASO  in  each  medical  center  has  access  to  an 
alphabetical  listing  of  the  staff  members  in  their  area  of 
responsibility.  In  order  to  distribute  surveys  at  the 
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assigned  rate,  the  MSO  points  of  contact  were  asked  to  give 
that  number  of  surveys  to  each  of  their  TASOs  equal  to  the 
medical  center's  distribution  rate  based  on  the  number  of 
staff  members  assigned  to  them.  The  TASOs  were  then 
instructed  to  distribute  one  survey  to  every  seventh  staff 


TABLE  3 

SURVEY  DISTRIBUTION  AND  RETURN  RATES 


Medical  Center 

#  Surveys 

#  Users 

Dist.  Rate 

Response  Rate 

% 

Andrews 

230 

1600 

1  per  7 

149/230 

64.8 

Keesler 

290 

2000 

1  per  7 

111/290 

38.3 

Lackland 

450 

4500 

1  per  10 

135/450 

30.0 

Scott 

230 

1000 

1  per  5 

124/230 

53.9 

Travis 

230 

1600 

1  per  7 

94/230 

40.9 

Wright-Patterson 

260 

1800 

1  per  7 

103/260 

39.6 

member  on  their  list  (every  fifth  staff  member  at  Scott  and 
every  tenth  staff  member  at  Lackland.)  An  example  of  the 
distribution  instructions  is  provided  at  Appendix  E. 

This  method  of  distribution  is  called  systematic  sam¬ 
pling  and  virtually  ensured  that  every  medical  department, 
occupational  type,  rank,  and  computer  experience  level  would 
be  represented.  Further,  by  choosing  every  fifth,  seventh, 
or  tenth  person  on  each  TASO's  list,  a  high  degree  of  ran¬ 
domness  was  provided  (Emory  and  Cooper , 1991 : 265 ) . 

Survey  Follow-Up  Procedures.  The  CIOs  at  the 
medical  centers  were  very  helpful  in  assisting  with  follow¬ 
up  procedures  after  the  system  user  surveys  were  distri¬ 
buted.  Specifically,  the  CIOs  at  Keesler,  Andrews,  and 
Scott  enclosed  their  own  cover  letters  with  each  survey 
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encouraging  participation.  Additionally,  electronic  mail 
messages,  originating  at  the  MSO,  were  transmitted  at 
Keesler,  Scott,  Andrews,  and  Lackland  as  reminders  to  all 
recipients  to  fill  out  and  return  the  surveys  as  soon  as 
possible.  The  points  of  contact  at  Andrews,  Travis,  and 
Scott  personally  spoke  to  all  of  the  medical  center's  TASOs 
to  encourage  them  to  follow-up  within  their  own  areas. 

Prior  to  survey  distribution  at  Wright-Patterson,  the  CIO 
made  announcements  at  staff  meetings  about  the  survey  and 
explained  the  importance  of  high  response  rates  and  accurate 
completion  of  the  answer  sheets.  After  the  surveys  were 
distributed  at  Travis  and  Wright-Patterson,  written  remind¬ 
ers  were  sent  to  recipients  to  encourage  participation. 

A  delay  in  the  distribution  of  surveys  at  Travis 
resulted  from  a  misunderstanding  concerning  survey  proce¬ 
dures.  Immediately  after  the  issue  was  resolved,  the  sur¬ 
veys  were  distributed  and  the  point  of  contact  diligently 
folio,  ed  up  with  recipients. 

Further,  the  researchers  remained  in  telephone  contact 
with  the  medical  center  points  of  contact  to  promote  rapid 
and  high  response  rates.  Notwithstanding  the  transmission 
errors  inherent  in  electronic  mail  systems  and  possibility 
of  paper  memos  not  reaching  the  intended  recipient  in  mail 
distribution  systems,  each  person  to  receive  a  user  survey 
at  each  medical  center  was  ideally  contacted  at  least  two 
times . 
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By  24  July  1992,  a  total  of  718  usable  surveys  had  been 
returned  to  the  researchers  for  an  overall  response  rate  of 
42.5  percent.  Not  counted  in  the  response  rate  were  21 
unusable  surveys,  either  because  most  of  the  data  was  miss¬ 
ing  or  the  demographic  data  was  inconsistent  (e.g.,  three 
answers  to  the  same  question.) 

Instrument  Validity.  Validity  of  measurement  is 
"the  ability  of  a  research  instrument  to  measure  what  it  is 
purported  to  measure"  (Emory  and  Cooper , 1991 : 180 ) .  Emory 
and  Cooper  (1991)  further  state  that  instrument  validity  can 
be  classified  into  three  major  types:  content  validity, 
criterion-related  validity,  and  construct  validity. 

Criterion-related  validity  is  the  level  of  correlation 
between  the  results  of  the  measurement  instrument  and  some 
set  of  external  criteria  (Emory  and  Cooper , 1991 : 181-182 ) . 
Clearly,  at  this  stage  of  research,  there  have  been  no 
criterion  measures  developed. 

Construct  validity  is  the  degree  to  which  the  measure¬ 
ment  instrument  is  related  to  the  overall  theoretical  frame¬ 
work.  A  construct  is  an  abstract  idea  which  is  not  directly 
measurable.  Evidence  of  construct  validity  normally 
involve*’  multiple  measures  of  the  constructs  under  consider¬ 
ation  (Emory  and  Cooper , 1991 : 182-184 ) .  The  current  study’s 
design  makes  no  provision  for  such  extra  measures.  Thus,  it 
was  not  possible  to  validate  Garvin's  dimensions  of  quality 
constructs  in  this  study. 
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Content  validity  is  the  type  most  applicable  to  sur¬ 
veys.  An  instrument  is  said  to  have  content  validity  if  it 
provides  sufficient  coverage  of  the  research  topic.  It  is 
usually  a  judgmental  determination  by  experts.  Specifi¬ 
cally,  the  instrument  is  evaluated  in  terms  of  how  well  it 
covers  all  aspects  of  the  topic,  and  the  individual  survey 
items  are  evaluated  for  adequacy  and  correctness.  Content 
validity  cannot  be  established  quantitatively  (Emory  and 
Cooper, 1991 : 180) .  The  content  validity  of  the  user  survey 
was  determined  through  pretest  and  assessment  by  locally 
av-ailable  Medical  Service  Corps  officers  and  other  medical 
systems  experts.  The  pretest  process  was  described  earlier 
in  this  chapter. 

An  instrument's  reliability  is  the  degree  to  which  it 
provides  consistent  results,  that  is  its  dependability, 
stability,  and  predictability.  Reliability  is  a  necessary 
contributor  to,  but  not  sufficient  for  the  establishment  of 
validity.  Reliability  can  be  measured  using  several  differ¬ 
ent  approaches.  Some  of  these  approaches  are  test-retest 
(stability),  parallel  forms  (equivalence),  split-half,  and 
internal  consistency  (Emory  and  Cooper , 1991) .  In  the  system 
user  survey,  the  internal  consistency  approach  was  used  to 
evaluate  critical  quality  factors. 

Cronbach's  alpha  coefficient  was  used  to  assess  the 
level  of  reliability  among  the  questions  that  sought  to 
evaluate  the  overall  quality  of  the  MSO  staff  and  the 
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available  information  systems.  Cronbach's  alpha  coefficient 
was  useful  in  determining  the  consistency  of  responses  to 
these  questions.  According  to  Cronbach,  alpha  is  "an  esti¬ 
mate  of  the  correlation  between  two  random  samples  of  items 
from  a  universe  of  items  like  those  in  the  test"  (Cronbach, 
1951:297).  The  correlation  between  any  two  questions  seek¬ 
ing  to  measure  the  same  characteristic  is  also  called  a 
coefficient  of  equivalence,  "showing  how  nearly  two  measures 
of  the  same  general  trait  agree"  (Cronbach, 1951 : 298 ) .  The 
formula  for  calculating  Cronbach's  alpha  coefficient  is 
n/(n-l)  times  the  ratio  of  interitem  covariance  to  total 
variance  where  n  is  the  number  of  items  (Cronbach, 1951 ) . 

The  researchers  used  SAS,  a  statistical  analysis  computer 
application,  to  calculate  the  alpha  coefficients  (SAS  Insti¬ 
tute, 1990a  and  1990b). 

In  formulating  his  equation  for  alpha  as  a  measure  of  a 
test's  reliability,  Cronbach  gave  a  conservative  estimate  of 
at  least  0.6  as  indicating  adequate  reliability  especially 
since  alpha  is  usually  an  underestimate  of  actual  interitem 
correlation  (Cronbach, 1951 ) .  The  researchers  chose  to  use 
0 . 6  as  the  minimum  level  for  interitem  reliability  in  this 
study . 

Some  concepts,  specifically  the  users’  opinions  of 
information  system  quality  and  their  levels  of  satisfaction 
with  information  system  services,  were  incorporated  into  the 
survey  usir\g  multiple  questions  to  formulate  an  answer  to 


75 


Investigative  Question  5:  To  what  extent  do  the  customers 
feel  that  the  Medical  Systems  Office  is  providing  them  with 
quality  technical  support  in  information  systems? 

Cronbach's  alpha  coefficient  was  used  to  determine  if 
respondents  interpreted  questions  as  intended  by  the 
researchers.  This  reliability  measure  was  applied  to  the 
multiple  questions  to  used  build  each  composite  response. 
Table  4  lists  the  variables  determined  by  the  answers  to 
multiple  questions  and  the  survey  question  numbers  pertain¬ 
ing  to  them.  The  reliability  of  each  variable  when  using 
all  components  is  also  listed.  Each  variable  was  shown  to 
be  reliable  with  an  acceptable  Cronbach  alpha  score. 

TABLE  4 

VARIABLES  DETERMINED  BY  MULTIPLE 
USER  SURVEY  QUESTIONS 

Variable  Description  Survey  Question  #  Cronbach  Alpha 

1  User  confidence  level  30,32,58,59  0.620535 

with  information 

system  and  MSO  office 
staff 

2  User  satisfaction  34,35,36,40,41,  0.869922 

level  with  informa-  42,46,47,48,52, 

tian  system  53,54,60 

Variable  one  is  a  consolidation  of  the  responses  to 
questions  asking  the  respondent  to  rate  the  level  in 
improvement  in  the  user's  quality  of  work  as  a  result  of 
computers,  the  level  of  confidence  the  user  has  in  the 
medical  center's  computers,  the  level  of  confidence  the  user 
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has  in  the  medical  center's  MSO  staff,  and  the  level  of 
satisfaction  the  user  has  with  assistance  provided  by  the 
MSO  staff.  This  variable's  alpha  coefficient  was  acceptable 
by  the  criterion  presented  above. 

Variable  two  was  calculated  from  several  sets  of 
related  questions.  One  set  of  questions  concerning  the 
user's  level  of  satisfaction  with  performance,  additional 
features,  serviceability,  and  aesthetics  were  asked  three 
times  so  that  the  respondent  could  evaluate  up  tc  three 
different  individual  information  systems  in  these  areas. 
Further,  another  set  of  questions  asked  the  user  to  rate  the 
level  of  importance  he  or  she  considers  applicable  to  each 
of  the  above  areas.  These  importance  ratings  were  used  to 
weight  the  above  satisfaction  ratings.  Each  evaluated 
system's  satisfaction  factor  importance  score  was  converted 
to  a  decimal  with  a  score  of  2  (very  low  importance)  equal 
to  0.2  and  a  score  of  6  (very  high  importance)  equal  to  1. 

As  discussed  earlier,  a  score  of  1  (no  opinion)  was  consid¬ 
ered  to  be  a  nonresponse.  Each  system's  satisfaction  factor 
score  was  then  multiplied  by  its  importance  weight  to  get  a 
weighted  satisfaction  score.  These  weighted  scores  were 
used  to  calculate  the  reliability  for  user  satisfaction  with 
information  system.  In  addition,  the  user's  satisfaction 
with  the  overall  facility-wide  information  system  was  con¬ 
sidered.  The  alpha  coefficient  for  variable  two  was 


77 


sufficiently  high  to  ensure  high  confidence  in  results  from 
this  measure. 

Statistical  Tests.  Upon  receipt  of  the  completed 
user  surveys,  the  results  were  sorted  by  medical  center  and 
loaded  into  six  separate  data  files  along  with  a  file 
containing  the  total  data  set.  Each  answer  sheet  was  veri¬ 
fied  for  accuracy  and  rechecked  by  the  researchers  to  ensure 
the  answers  were  usable.  Several  statistical  analysis 
programs  were  written  in  SAS  for  the  purpose  of  examining 
the  data.  The  programs  were  thoroughly  pretested  using  a 
sample  set  of  data  (consisting  of  15  answer  sheets)  to 
ensure  that  the  programs  were  error-free  and  that  they 
accurately  measured  the  frequencies  and  interitem  correla¬ 
tions  as  intended. 

A  Chi-Square  ( X 2)  test  was  administered  to  determine  if 
the  distribution  of  the  responses  among  the  six  research 
sites  differed  from  the  expected  distribution  based  on  the 
overall  response  percentage  rate.  Any  significant  differ¬ 
ences  between  the  actual  site  response  rate  and  that 
expected  would  indicate  unequal  or  skewed  representation 
across  the  six  medical  centers  (McClave  and  Benson , 1991 ) . 

The  researchers  received  718  usable  responses  out  of 
1690  surveys  distributed,  resulting  in  a  response  rate  of 
42.5  percent.  A  Chi-Square  test  was  done  to  determine  if 
the  distribution  of  responses  significantly  differed  from 
the  number  of  expected  responses  by  medical  center.  The 
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researchers  felt  this  test  was  necessary  to  ensure  an 
unskewed  representation  across  medical  centers.  The  null 
hypothesis  for  the  procedure  was  that  the  responses  were  not 
biased  across  medical  centers.  The  calculation  revealed 
that  the  response  rates  were,  in  fact,  skewed  at  an  alpha  of 
0.05,  with  a  Chi-Square  of  51.6,  leading  the  researchers  to 
reject  the  null  hypothesis. 


TABLE  5 


OBSERVED  VERSUS  EXPECTED  MEDICAL  CENTER  RESPONSE 


Medical  Center  Observed 

Andrews  149 

Keesler  111 

Lackland  135 

Scott  124 

Travis  94 

Wright-Patterson  103 


Expected 
98 
123 
191  . 
98 
98 
110 


Table  5  shows  the  actual  number  of  surveys  received 
(observed)  and  the  number  expected,  based  on  an  overall 
response  rate  of  42.5  percent,  for  each  survey  site.  The 
medical  centers  at  Andrews  AFB  and  Lackland  AFB  were  the 
most  significant  contributors  to  the  high  Chi-Square  value. 
The  expected  response  frequency  for  Andrews  was  98  while  its 
observed  frequency  was  149.  The  expected  response  frequency 
for  Lackland  was  191  while  its  observed  frequency  was  135. 

Since  the  responses  (particularly  from  Andrews  AFB  and 
Lackland  AFB)  were  not  consistent  with  uniform  representa¬ 
tion  across  medical  centers,  some  caution  must  be  taken  in 
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interpreting  results.  This  subject  will  be  discussed  fur¬ 
ther  in  Chapter  V. 

SAS  programs  calculated  the  frequencies,  means,  and 
standard  deviations  of  each  of  the  user  survey  questions. 

In  addition,  a  Spearman's  rho  rank-correlation  coefficient 
was  calculated  to  measure  the  strength  of  the  correlation 
among  the  user’s  educational  level,  prior  computer  training, 
and  level  of  satisfaction  with  the  medical  center's  computer 
systems.  All  of  the  above  values  were  used  to  make  further 
conclusions  summarized  in  Chapters  IV  and  V. 

Finally,  the  specific  research  question  (i.e.,  To  what 
extent  is  there  a  relationship  between  the  source  of  MSO 
staffing  and  level  of  quality  of  the  information  system  in 
United  States  Air  Force  Medical  Centers?)  was  answered  with 
a  measure  of  the  strength  of  the  correlation  between  staff¬ 
ing  source  (from  the  CIO  survey)  and  the  quality  of  the 
information  services  (from  the  user  survey)  provided  at  each 
medical  center. 

Summary 

This  chapter  has  provided  a  description  of  the  research 
process  for  this  study.  Four  seprrate  phases  of  research 
have  been  identified  and  discussed.  Phase  One  involved  a 
review  of  the  existing  literature  on  trends  in  hospital 
information  systems  and  the  standards  and  goals  of  medical 
information  services  in  the  Air  Force.  The  results  of  this 
research  phase  were  reported  in  Chapter  I. 
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This  chapter  focused  its  discussion  on  the  final  three 
phases  of  the  study.  Phase  Two  involved  the  development  and 
administration  of  a  survey  instrument  for  the  CIO  at  each  of 
the  six  Air  Force  medical  centers.  The  purpose  of  this 
phase  was  to  gather  information  pertaining  to  MSO  staffing 
and  available  information  systems  at  each  medical  center. 

The  population  of  interest,  measurement  instrument,  develop¬ 
ment  of  the  instrument,  and  instrument  pretesting  method 
were  discussed.  In  addition,  the  data  collection  plan, 
survey  follow-up  procedures,  and  data  analysis  design  were 
outlined . 

Phase  Three's  objective  was  to  gather  data  about  the 
information  systems  most  prevalent  at  Air  Force  medical 
centers  in  order  to  provide  the  researchers  with  a  more 
objective  basis  for  comparison  among  the  six  research  sites. 
Brief  descriptions  of  major  information  systems  are  in 
Chapter  IV. 

Finally,  Phase  Four  involved  the  development  and  admin¬ 
istration  of  a  survey  instrument  to  be  given  to  a  sample  of 
information  system  customers  at  the  six  medical  centers. 

The  purpose  of  this  survey  was  to  quantitatively  determine 
the  level  of  customer  satisfaction  with  the  available 
information  systems  and  services  provided  by  the  MSO  staff. 
The  population  of  interest  for  this  survey,  the  measurement 
instrument,  and  the  instrument  development  procedures  were 
described.  Instrument  pretesting,  plan  for  data  collection. 
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follow-up  procedures,  and  instrument  validity  were  also 
discussed.  Finally,  the  plan  for  analyzing  the  data  gath¬ 
ered  from  the  administration  of  the  user  survey  was 
reported.  Chapter  IV  discusses  the  findings  of  these  analy¬ 
ses.  The  researchers'  recommendations  and  conclusions  based 
on  the  findings  are  found  in  Chapter  V. 
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IV.  ANALYSIS  OP  DATA 


Introduction 

This  chapter  discusses  and  analyzes  the  data  gathered 
in  Phases  Two,  Three,  and  Four  of  the  research.  Phase  Two 
involved  the  development  and  administration  of  a  CIO  survey 
which  was  used  to  gather  factual ,  statistical  data  about 
operations,  staffing,  and  personnel  in  each  MSO.  In  Phase 
Three,  the  researchers  conducted  telephone  interviews  with 
medical  systems  experts  at  AFMSA  to  obtain  detailed 
descriptions  of  the  major  information  systems  at  the  six 
medical  centers.  Phase  Four  involved  the  development  and 
administration  of  a  User  survey  designed  to  determine  the 
users'  level  of  satisfaction  with  the  MSO  and  information 
systems  at  their  medical  centers. 

First,  there  is  a  discussion  of  the  responses  to  the 
CIO  survey  from  Phase  Two.  This  section  describes  the 
individual  MSO  demographics  to  include  a  depiction  of 
staffing  characteristics  at  each  medical  center  followed  by 
a  listing  of  available  information  systems  at  the  six 
research  sites. 

Next,  the  results  of  the  system  user  survey  are 
detailed.  In  this  discussion,  an  assessment  of  respondent 
representativeness  and  a  summary  of  respondent  demographics 
are  included.  The  user  responses  are  analyzed  by  medical 
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center  and  then  by  individual  information  system.  Then,  an 
overall  picture  of  all  responses  is  drawn. 

Finally,  the  functions,  capabilities,  and  other  fac¬ 
tors  of  the  major  information  systems  found  at  Air  Force 
medical  centers  are  summarized. 

This  chapter  answers  Investigative  Questions  1,  4, 
and  5  which  follow. 

Question  1.  What  is  the  source  of  each  Medical  Sys¬ 
tems  Office  staff  member  at  each  medical  center  (i.e., 
in-house,  other  military  organization,  or  civilian  organi¬ 
zation)? 

Question  4.  To  what  extent  are  the  Medical  Systems 
Offices  meeting  the  standards  and  goals  set  by  the  Air 
Force  for  the  operation  of  medical  information  systems? 

Question  5.  To  what  extent  do  the  customers  feel  that 
the  Medical  Systems  Office  is  providing  them  with  quality 
technical  support  in  information  systems? 

The  CIO  Survey 

This  section  presents  the  results  of  the  CIO  survey 
found  at  Appendix  C  and  serves  to  answer  Investigative 
Questions  1  and  4. 

Question  1.  What  is  the  source  of  each  MSO  staff 
member  at  each  medical  center,  (i.e.,  in-house,  other  mili¬ 
tary  organization,  or  civilian  organization)? 
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Question  4.  To  what  extent  are  the  Medical  Systems 
Offices  meeting  the  standards  and  goals  set  by  the  Air 
Force  for  the  operation  of  medical  information  systems? 

The  CIO  survey  was  sent  to  the  chief  information  offi¬ 
cers  at  each  of  the  six  USAF  medical  centers  in  the  conti¬ 
nental  United  States.  The  survey  was  a  fill-in-the  blank 
format,  designed  to  collect  factual,  statistical  data  about 
operations,  staffing,  and  personnel  from  each  medical  sys¬ 
tems  office. 

All  six  of  the  CIOs  responded  to  the  survey  resulting 

in  a  response  rate  of  100  percent.  All  of  the  surveys  were 

usable  and  are  included  in  the  analysis. 

Medical  System  Office  Personnel  Functions.  Table  6 

presents  a  summary  of  the  number  of  personnel  and  the 

personnel  functions  in  the  information  systems  offices  as 

reported  in  the  CIO  surveys.  In  Table  6  and  all  succeeding 

tables  and  figures,  the  names  of  the  medical  centers  are 

abbr e vi a t ed  as  foil ows : 

David  Grant  Medical  Center  =  DGMC 
Keesler  Medical  Center  =  KMC 
Malcolm  Grow  Medical  Center  =  MGMC 
Scott  Medical  Center  =  SMC 
Hilford  Hall  Medical  Center  =  WHMC 
Wright-Patterson  Medical  Center  =  WPMC. 
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TABLE  6 

MEDICAL  SYSTEM  OFFICE  PERSONNEL  FUNCTIONS 


NAME  OB' 

MEDICAL 

CENTER  1 

STAFF  FUNCT I ONS 

DGMC 

KMC 

cup 

WHMC 

WPMC 

Programers 

2 

4 

1C 

5 

Coiputer  Operators 

WM 

7 

1 

7 

w 

Praising/ lark  Center  Support 

5 

2 

Adninistrative  Support 

2 

5 

Composite  Health  Care  Spstea  (C1CS) 

Support 

4 

2 

Individual  System  Specialists 

7 

2 

1 

8 

*  * 

LAI  Specialists 

7 

1 

* 

Custoner  Service 

2 

2 

12 

4 

Hasageaent 

1 

2 

2 

1 

9 

3 

luiber  on  IS  Staff 

12 

30 

16 

7 

i 

51 

32 

There  is  a  wide  range  in  the  number  of  personnel  working 
in  each  MSO.  As  shown  in  Table  5,  the  number  of  personnel 
ranged  from  a  low  of  seven  at  Scott  AFB  to  a  high  of  fifty- 
one  at  Wilford  Hall. 

The  surveys  revealed  that  personnel  in  the  MSOs  work 
in  one  of  nine  functional  areas.  These  nine  areas  are: 
programmers,  computer  operators,  training/work  center 
support,  administrative  support.  Composite  Health  Care 
System  (CHCS)  specialists,  individual  system  specialists, 
local  area  network  (LAN)  specialists,  customer  service,  and 
management . 
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CHCS  is  a  new  medical  system  currently  being  fielded 
for  outpatient  use  throughout  DOD.  Presently,  Keesler  APB 
and  Malcolm  Grow  Medical  are  using  CHCS  for  outpatient 
clinics  and  serving  as  inpatient  test  sites  for  the  system. 

Medical  System  Office  Personnel  Staffing  by  Source. 
Table  7  presents  a  summary  of  the  source  of  staffing  for 
information  systems  office  personnel .  The  sources  of 
staffing  are  divided  into  four  categories.  USAF  medical 
systems  refers  to  any  active  duty  Air  Force  member  with  a 
medical  systems  specialty.  Their  experience  and  training 
requirements  are  specifically  stated  in  Air  Force  Regula¬ 
tion  39-1  and  are  authorized  to  carry  a  "W"  prefix  on  their 
Air  Force  Specialty  Code  (AFSC)  which  denotes  their  duty 
qualifications  and  experience.  They  are  medical  special¬ 
ists  who  have  undergone  extra  training  in  computer  applica¬ 
tion  design,  analysis,  supervision,  and  monitoring  and  who 
have  held  a  medical  systems  job  position  for  at  least  24 
months  (AFR  39-1). 

Personnel  with  an  "SC”  specialty  would  normally  work 
in  a  communications  squadron.  USAF  SC  personnel  do  not 
receive  any  formal  training  in  hospital  administration, 
operations  or  medical  systems.  Rather,  they  are  trained  in 
the  operation  of  generic  information  systems. 
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A  civilian  government  employee  is  a  USAF  General 
Schedule  (GS)  or  a  General  Management  (GM)  employee  working 
in  a  medical  systems  related  occupational  code  title. 

TABLE  7 


MEDICAL  SYSTEM  OFFICE  PERSONNEL  STAFFING  BY  SOURCE 


NAME  OB'  MEDICAL  CENTER  | 

SOURCE  OF  STAFF 

DGMC 

KMC 

MGMC 

SMC 

WHMC 

WPMC 

Military  Medical  Systems 

5 

4 

C 

* 

6 

Military  SC 

3 

4 

4 

15 

Civilian  Government 

Employee 

6 

6 

7 

6 

29 

5 

Contractor 

17 

3 

13 

2 

Other 

1 

1 

w 

writ 

12 

30 

IS 

■a 

51 

32 

lumber  Supervised  by  Medical  Center 

12 

.  13 

0 

0 

0 

IS 

Percent  Supervised  by  Medical  Center 

loot 

43% 

Sit 

loot 

751 

56% 

luiber  Supervised  by  SC 

0 

0 

0 

n 

0 

12 

Percent  Supervised  by  SC 

01 

St 

mm 

ot 

wa 

38S 

Humber  Supervised  by  Contractor 

0 

17 

3 

0 

13 

2 

Percent  Supervised  by  Contractor 

01 

57t 

ist 

ot 

25% 

6t 

A  contractor  is  a  civilian  employee  who  is  working  in 
an  information  systems  office  but  is  employed  by  a  commer¬ 
cial  firm  contracted  by  the  Air  Force. 

The  category  "Other"  refers  to  those  personnel  working 
in  the  MSO  who  are  medical  specialists  but  have  not  under¬ 
gone  extra  training  in  computer  application  design. 
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analysis,  supervision,  or  monitoring.  As  an  example,  at 
Scott  Medical  Center,  there  is  a  military  cardiopulmonary 
technician  working  in  the  MSO. 

The  categories  below  the  row  labelled  "TOTAL"  are  not 
categories  of  source  of  staffing.  "Number  supervised  by 
medical  center"  is  the  number  of  people  on  the  MSO  staff 
supervised  by  the  medical  center.  This  category  includes 
staff  members  who  are  military  medical  systems,  civilian  GS 
or  GM  employees,  others,  and  SC  personnel  who  are  super¬ 
vised  in  their  activities  by  the  MSO  staff.  "Number  super¬ 
vised  by  SC"  is  the  number  of  people  on  the  MSO  staff  who 
are  supervised  by  the  communications  squadron.  "Number 
supervised  by  contractor"  is  the  number  of  people  super¬ 
vised  by  the  contractor  by  which  they  are  employed.  All  of 
the  contractor  personnel  at  the  medical  centers  are  super¬ 
vised  by  a  representative  of  the  contractor  rather  than  the 
MSO.  When  dealing  with  issues  concerning  contract  person¬ 
nel,  the  MSO  staff  must  deal  with  a  government  contracting 
representative.  The  government  contracting  representative 
presents  the  issues  to  the  contractor's  representative  who 
then  deals  with  the  contractor  personnel.  The  only  person¬ 
nel  who  can  fall  into  this  category  are  contractor  person¬ 
nel  and  SC  personnel  who  are  not  supervised  by  the  MSO  and 
contractor  personnel . 

The  researchers  were  surprised  by  the  mix  of  personnel 
staffing  sources  at  all  of  the  medical  centers.  Table  7 
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reveals  that,  with  the  exception  of  David  Grant  and  Scott, 
each  medical  center  has  staff  members  in  at  least  three 
categories  of  staffing  source.  Prior  to  receiving  the 
results  of  the  CIO  surveys,  the  researchers  thought  it 
would  be  appropriate  to  categorize  each  medical  center  by 
primary  source  of  staffing,  (i.e.,  source  of  staffing  at 
one  medical  center  might  be  predominantly  civilian  contrac¬ 
tor).  However,  with  the  mix  of  personnel  staffing  sources 
at  individual  medical  centers,  the  researchers  found  that 
it  would  be  misleading  to  attempt  to  categorize  each  medi¬ 
cal  center  by  source  of  staffing. 

\ 

Nonetheless,  Figures  5  and  6  show  that  some  general¬ 
izations  can  be  made  from  the  CIO  survey  data  concerning 
MSO  staffing  sources.  In  both  figures,  the  names  of  the 
medical  centers  are  abbreviated  as  follows: 

David  Grant  Medical  Center  =  DGMC 
Keesler  Medical  Center  =  KMC 
Malcolm  Grow  Medical  Center  =  MGMC 
Scott  Medical  Center  -  SMC 
Nil  ford  Hall  Medical  Center  =  WHMC 
Wright-Patterson  Medical  Center  =  WPMC. 

First,  the  Scott  and  David  Grant  Medical  Center  MSOs 

have  no  contracted  or  SC  staff  members.  They  are  made  up 

entirely  of  military  medical  and  civilian  government 

employees.  On  the  other  hand,  Keesler’s  MSO  is  staffed 

about  50  percent  by  contracted  personnel ,  far  more  than  any 

other  medical  center. 
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Figures.  MSO  Staffing  by  Source 


The  David  Grant,  Hilford  Hall,  and  Scott  MSOs  have  50 
percent  or  more  civilian  government  employees.  Scott's 
civilian  government  work  force  makes  up  about  85  percent  of 
the  total . 

Finally,  Wright-Patterson  MSO  has  the  largest  propor¬ 
tion  of  SC  staff  members  of  all  the  medical  centers  with 
about  50  percent  of  its  staff  made  up  of  SC  personnel . 

There  is  a  smaller  proportion  of  SC  employees  on  the  MSO 
staffs  at  Keesler,  Malcolm  Grow,  and  Wilford  Hall. 

Malcolm  Grow  has  the  most  heterogeneous  staffing  make 
up,  with  each  of  the  four  main  categories  fairly  well 
represented. 

Further  observations  can  be  made  about  medical  center 
staffing  if  civilian  contractors  and  individuals  who  are 
supervised  by  SC  are  grouped  together  because  they  are  not 
supervised  by  medical  center  personnel .  Figure  6  shows 
percent  under  medical  center  supervision  versus  percent 
under  outside  supervision.  In  Figure  6,  the  category  "MC 
supervision"  refers  to  the  percentage  of  personnel  from 
Table  7  who  are  supervised  by  medical  center.  The 
category  "outside  supervision"  is  the  percentage  of  person¬ 
nel  who  are  either  supervised  by  SC  or  supervised  by  a 
contractor.  Most  noteworthy  are  the  facts  that  almost  57 
percent  of  Keesler's  MSO  staff  and  almost  44  percent  of 
Wright-Patterson's  MSO  staff  are  under  outside  supervision. 


92 


Medical  Center  Information  Systems.  Prom  the  CIO 
surveys,  the  researchers  obtained  a  list  of  information 
systems  currently  operating  at  each  medical  center.  Table 
8  is  a  consolidation  of  these  lists  and  was  developed  to 
provide  a  roster  of  information  systems  in  all  of  the 
medical  centers  and  to  show  which  medical  centers  operate 
each  system.  A  more  detailed  description  of  the  functions 
and  capabilities  of  the  major  systems  is  found  in  the  next 
section  of  this  chapter  and  at  Appendix  F. 
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TABLE  8 


MEDICAL  CENTER  INFORMATION  SYSTEMS 


NAME  OF  MEDICAL  CENTER 

INTORMAT X ON 

SYSTEM 

DGMC 

KMC 

MGMC 

SMC 

WHMC 

WFMC 

Autonated  Quality  of  Care 
Evaluation  Support  Systen 
(4QCESS) 

I 

! 

Z 

Z 

z 

Z 

Conanding  Medical  Inforution 
Systen  (OHS) 

Z 

Composite  Health  Care  Systen 
(CHCS)  * 

l 

Z 

Conputer  Assisted  Processing  of 
Cardiogram  (CAPOC) 

Z 

z 

Z 

CPC  System 

z 

Cental  Data  Systen 

Z 

z 

zas-iii/meprs 

Z 

wm 

Z 

z 

z 

Hospital  Intonation  Systen 

z 

Managed  Health  Care 

z 

mm 

Z 

z 

i 

Z 

z 

z 

PC  III 

i 

z 

SAS  III 

z 

WPS  PC  Hetrork 

z 

THIPQQD 

i 

z 

z 

TRILAB 

! 

z 

z 

ES9HHH 

z 

z 

Z 

z 

z 

z 

TRIRAD 

z 

Z 

n 

z 

Hang  Inventory 

z 

*lote:  Sfstea  is  approved  and  is  being  fielded  fs;  outpatient  use  throughout  the  OQD. 

It  is  being  tested  for  inpatient  use  at  Keesler  and  Malcolm  Grot  Medical  Centers. 
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Medical  Center  Control  Statistics.  There  was  one 
common  link  in  the  control  statistics  which  the  medical 
centers  track  and  report.  All  of  the  medical  centers 
reported  that  they  track  the  amount  of  downtime  for  their 
major  information  systems  on  a  monthly  basis.  As  the  down¬ 
time  is  compiled  and  recorded/  it  is  reported  as  System 
Availability  or  percentage  of  time  the  systems  were  opera¬ 
tional.  At  David  Grant,  Keesler,  Malcolm  Grow,  and  Scott 
Medical  Centers,  System  Availability  was  the  only  control 
statistic  reported  by  the  MSO. 

The  MSOs  at  the  larger  medical  centers  reported  that 
they  track  additional  control  statistics.  Wilford  Hall 
reports  three  additional  control  statistics.  The  MSO  per¬ 
forms  a  random  system  monitor  of  software  at  three-day 
intervals.  This  measurement  tracks  Central  Processing  Unit 
(CPU)  response  time  and  indicates  if  response  time  for  a 
system  is  slow.  A  daily  error  log  is  kept  to  record  system 
errors,  and  a  VAX  Performance  Advisor  measurement,  a  soft¬ 
ware  product  of  VAX,  is  recorded  to  monitor  hardware  and 
software  system  performance  at  the  time  of  a  system  problem 
and  quarterly. 

The  Wright-Patterson  MSO  tracks  trouble  response  time 
and  trouble  repair  time.  Trouble  response  time  is  the 
amount  of  time  it  takes  for  a  service  technician  to  respond 
to  a  reported  system  problem,  and  trouble  repair  time  is 
the  amount  of  time  that  it  takes  to  repair  a  system  problem 
from  the  time  a  service  technician  responds  to  a  reported 


95 


system  problem.  In  addition,  the  MSO  conducts  an  annual 
survey  among  the  information  system  users  in  the  medical 
center.  The  survey  allows  users  to  let  the  MSO  know  in 
which  areas  the  users  desire  additional  applications, 
services,  and  training.  These  results  are  analyzed  and 
reported  by  the  MSO  to  the  medical  center  commander  and  to 
the  users.  The  MSO  also  conducts  monthly  training  classes 
for  users  in  applications  such  as  Enable  and  Wang  E-.uai  1 . 
The  attendance  at  these  classes  is  reported  as  a  control 
statistic. 

The  Pser  Survey 

A  total  of  1,690  user  surveys  as  shown  at  Appendix  D 
were  distributed  to  information  system  users  at  the  six 
medical  centers.  718  usable  questionnaires  were  returned 
to  the  researchers,  for  a  42.5  percent  response  rate.  The 
purpose  of  this  survey  was  to  determine  the  users'  level  of 
satisfaction  with  the  MSO  and  information  systems  at  their 
medical  centers.  Additionally,  they  were  asked  to  rate  the 
performance,  additional  features,  serviceability,  and 
aesthetics  of  the  information  systems  they  use  most  often. 
The  information  gathered  from  this  survey  was  used  to 
answer  Investigative  Question  5:  To  what  extent  do  the 
customers  feel  that  the  medical  systems  office  is  providing 
them  with  quality  technical  support  in  information  systems? 
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Information  Systems  (Jser  Respondent  Demographics. 

There  were  739  respondents  in  the  survey  population  of 
1690,  resulting  in  a  response  rate  of  43.7  percent.  Of 
these  739  responses,  21  were  deemed  unusable  because  they 
were  either  filled  out  incompletely  or  contained  inconsis¬ 
tent  demographic  data  (e.g.,  one  respondent  indicated  that 
he  was  an  enlisted  member,  an  officer,  and  a  civilian 
employee) . 

After  eliminating  the  unusable  responses,  there  were 
718  usable  responses  which  resulted  in  an  adjusted  response 
rate  of  42.5  percent.  The  response  rates  by  medical  center 
are  shown  in  Table  3  in  Chapter  III.  Based  on  previous 
studies  done  by  Constantian  (1990)  and  Parker  (1987),  the 
researchers  expected  a  40%  response  rate. 

The  frequencies  and  percentages  of  the  key  demographic 
variables  from  the  survey  responses  follow.  In  the  Tables 
which  follow  in  this  section,  the  number  of  responses,  n, 
does  not  always  equal  718  because  some  survey  respondents 
did  not  answer  all  questions. 

Gender .  The  gender  of  the  respondents  was  almost 
equally  divided  with  355  female  respondents  and  357  male 
respondents.  Six  respondents  did  not  indicate  their  sex  on 
the  response  sheet. 

Aae.  The  typical  survey  respondent  was  in  one  of 
two  age  brackets:  either  26  to  32  years  old  (27.2  percent) 
or  33  to  39  years  old  (27.7  percent). 
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Education.  All  but  46  of  the  respondents  had  at 
least  a  high  school  diploma  as  their  highest  level  of 
education.  The  typical  survey  respondent  had  at  least  a 
high  school  diploma  (35.2  percent)  while  34.8  percent  of 
the  respondents  had  a  bachelor's  degree  or  higher. 

Status.  The  great  majority  of  respondents  were 
active  duty  military  (72.4  percent).  Civilian  civil  ser¬ 
vice  employees  accounted  for  24.2  percent  of  the  responses 
and  the  remaining  responses  (3.4  percent)  were  classified 
as  "other  civilian,"  such  as  volunteers. 

Grade .  There  were  530  military  respondents. 

The  complete  distribution  of  their  grades  is  shown  in 
Table  9. 

Of  the  169  civilian  civil  service  respondents,  the 
typical  grade  range  was  GS-5  to  GS-8.  A  complete  distribu 
tion  of  the  grades  of  civilian  civil  service  respondents  i 
shown  in  Table  10. 

There  were  twenty  civilian  respondents  who  were  not 
civilian  civil  service  employees.  The  majority  of  these 
were  contracted  employees. 
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TABLE  9 


GRADES  HELD  BY  MILITARY  RESPONDENTS  [n=530] 


Grade 

No.  of  Responses 

Percentaae 

E-l  to  E-4 

137 

25.8 

E-5  to  E-6 

143 

27.0 

E-7  to  E-9 

66 

12.5 

0-1  to  0-3 

84 

15.8 

0-4  to  0-5 

83 

15.7 

0-6  or  higher 

17 

3.2 

GRADES  HELD 

TABLE  10 

BY  CIVIL  SERVICE  RESPONDENTS 

[n=169] 

Grade 

No.  of  Responses 

Percentaae 

Wage  Grade 

2 

1.2 

GS-1  to  GS-4 

41 

24.3 

GS-5  to  GS-8 

93 

55.0 

GS-9  to  GS-12 

33 

19.5 

Corps.  This  cateaorv  shows  the  corps  to  which 

the  respondents  who 

are  active  duty  officers  are  assigned. 

The  Nurse  Corps  accounted  for  43.2  percent  of  the  active 

duty  officer  respondents.  A  complete  distribution  of 

active 

duty  officer 

respondents  is  shown 

below  in  Table  11 

TABLE  11 

OFFICER 

RESPONDENTS  BY  CORPS 

[n=183  ] 

Corps 

No.  of  Responses 

Percentaae 

Medical 

40 

21.9 

Dental 

9 

4.9 

Nurse 

79 

43.2 

Medical 

Service 

18 

9.8 

Biomedical  Service 

36 

19.7 

Other 

1 

.5 
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Duties .  There  was  a  complete  representation  of 
hospital  functions  among  the  survey  respondents.  About  one 
third  of  the  respondents  performed  some  type  of  medical 
administration  duties.  Table  12  shows  the  distribution  of 
duties  performed  by  the  respondents. 


TABLE  12 


DUTIES  OF  RESPONDENTS  [n=682] 


Duty  Location  No.  of  Responses  Percentage 


Direct  Inpatient  Care 
Direct  Outpatient  Care 
Ancillary  Service 
Medical  Administration 
Medical  Logistics 
Other 


139 

19.8 

121 

17.2 

105 

15.0 

228 

32.5 

40 

5.7 

69 

9.8 

Experience  Using  Computers.  Slightly  more  than  one 
third  of  the  respondents  had  between  five  and  ten  years  of 
experience  using  computers  either  on  or  off  the  job.  Only 
seven  percent  had  less  than  one  year  of  computer  experi¬ 
ence.  Table  13  summarizes  the  respondents'  experience 
using  computers. 
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TABLE  13 


EXPERIENCE  USING  COMPUTERS  [n=708] 


Experience 


No.  of  Responses 


Percentage 


Less  than  1  year 

50 

1  to  3  years 

145 

3  to  5  years 

175 

5  to  10  years 

256 

10  years  or  more 

82 

7.1 

20.5 
24.7 
36.2 

11.6 


Highest  Level  of  Computer  Training.  About  one 
third  of  the  respondents  taught  themselves  how  to  use  a 
computer  while  almost  eight  percent  had  not  received  any 
computer  training.  Another  one  third  reported  having 
completed  a  computer  course  at  either  the  high  school  or 
the  college  level,  2.8  percent  of  the  respondents  had  a 
degree  in  computer  science  ranging  from  an  associate  to  a 
master's  degree.  In  Table  14,  a  complete  distribution  of 
computer  training  is  shown  for  all  respondents. 


TABLE  14 

HIGHEST  LEVEL  OF  COMPUTER  TRAINING  [n=705] 


Level  of  Training  No.  of  Responses  Percentage 

None  54  7.7 

Self  Taught  234  33.2 

High  School  Course  46  6.5 

Adult  Education  129  18.3 

College  Course  188  26.7 

Associate  Degree  in  15  2.1 

Computer  Science 

Bachelor  Degree  in  4  .6 

Computer  Science 

Master's  Degree  or  higher  1  .1 

in  Computer  Science 

Other  34  4.8 
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Time  at  Current  Assignment.  The  range  of  time  at 
the  present  assignment  did  not  yield  any  unusual  results. 
Almost  twenty-five  percent  selected  the  most  frequent  cate¬ 
gory  of  "more  than  four  years."  This  may  seem  like  a  long 
time  for  active  duty  Air  Force  personnel,  but  almost  one 
fourth  of  the  respondents  were  civilian  employees  who  tend 
not  to  move  as  often  as  military  personnel.  A  summary  of 
job  experience  is  shown  in  Table  15. 


TABLE  15 


TIME  AT  CURRENT  ASSIGNMENT  [n=712] 


Time  at  Assignment 


No.  of  Responses  Percentage 


0  to  1  year 

1  to  2  years 

2  to  3  years 

3  to  4  years 
more  than  4  years 


159 

22.3 

150 

21.1 

135 

19.0 

96 

13.5 

172 

24.2 

Time  in  Current  Occupation.  There  were  no 
unusual  results  noted  in  the  range  of  tenure  in  current 
occupation  for  respondents.  The  distribution  of  time  in 
current  occupation  is  shown  in  Table  16. 


TABLE  16 


TIME  IN  CURRENT  OCCUPATION  [n=705] 


Time  in  Occupation 

No.  of  Responses 

Percentage 

0  to  5  years 

225 

31.8 

5  to  10  years 

157 

22.2 

10  to  15  years 

135 

19.1 

15  to  20  years 

108 

15.3 

more  than  20  years 

80 

11.3 
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Confidence  in  Osina  Computer  Applications.  The 
respondents  were  asked  to  rate  their  confidence  in  using 
nine  computer  applications.  They  rated  their  confidence 
for  each  application  using  the  following  Likert  scale 
intervals: 

1.  I  do  not  recognize  this  application. 

2.  I  do  not  use  it  at  all. 

3.  I  can  perform  only  basic  functions  following 
prompts  or  menus;  I  usually  need  help  recovering  from  mis¬ 
takes  . 

4.  I  can  perform  all  of  the  basic  functions  and 
follow  instructions  in  a  manual  for  more  advanced  func¬ 
tions;  I  sometimes  require  help  in  performing  more  advanced 
functions . 

5.  I  can  perform  all  of  the  basic  and  advanced  func¬ 
tions;  I  rarely,  if  ever,  require  assistance. 

6.  I  can  perform  all  of  the  functions  of  the  applica¬ 
tion;  others  seek  my  help  in  using  the  application. 

The  mean  ratings  for  the  applications  are  shown  in 
Table  17.  As  expected,  the  more  common  applications  such 
as  departmental  applications,  word  processing,  and  elec¬ 
tronic  mail  received  the  highest  scores.  Decision  support 
systems  which  have  only  recently  been  introduced  to  hospi¬ 
tals  received  the  lowest  ratings. 
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TABLE  17 


CONFIDENCE  RATINGS  IN  USING  COMPUTER  APPLICATIONS 


Application 

No.  of  Responses 

Mean  Ratinq 

Word  Processing 

713 

3.8 

Graphics 

706 

2.8 

Electronic  Mail 

704 

3.2 

Appointment  System 

703 

2.8 

Departmental  Applications 

698 

4.0 

Medical  Records  Access 

707 

2.5 

Report  Production 

710 

2.9 

Decision  Support  Systems 

709 

1.8 

Spreadsheets 

689 

2.5 

Satisfaction  Ratings.  Mean  satisfaction  ratings  for 
the  information  systems  were  calculated  from  all  usable 
responses  to  questions  34-57  as  shown  at  Appendix  D.  To 
calculate  satisfaction  ratings,  one  set  of  questions  con¬ 
cerning  the  user's  level  of  satisfaction  with  performance, 
additional  features,  serviceability,  and  aesthetics  was 
asked  three  times  so  that  the  respondent  could  evaluate  up 
to  three  different  information  systems  in  these  areas. 
Further,  another  set  of  questions  for  each  evaluated  system 
asked  the  user  to  rate  the  level  of  importance  for  each  of 
the  above  areas.  All  questions  that  surveyed  the  respon¬ 
dents'  opinions  were  based  on  a  5-point  Likert  scale.  The 
scale  of  responses  for  all  questions  requiring  a  subjective 
answer  was  considered  to  be  interval.  Although  the  respon¬ 
dents  had  six  answers  from  which  to  choose,  answer  number 
one  was  "No  opinion"  and  considered  to  be  a  nonresponse  and 
not  on  the  Likert  scale. 
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The  importance  ratings  were  used  to  weight  the  above 
satisfaction  ratings.  The  importance  score  of  each  evalu¬ 
ated  system's  satisfaction  factor  was  converted  to  a  deci¬ 
mal  with  a  score  of  2  (very  low  importance)  equal  to  0.2 
and  a  score  of  6  (very  high  importance)  equal  to  1.  Each 
system's  satisfaction  factor  score  was  then  multiplied  by 
its  importance  weight  to  get  a  weighted  satisfaction  score. 
These  satisfaction  ratings  are  presented  in  the  following 
three  ways  in  this  section.  They  are  shown  in  total  with¬ 
out  regard  to  information  system  or  medical  center,  by 
medical  center  for  information  systems  in  operation,  and 
individually  by  information  system  regardless  of  medical 
centers  in  which  it  is  operational . 

Total  Satisfaction  Ratings.  A  mean  satisfaction 
rating  was  calculated  across  all  medical  centers  for  all 
information  systems.  Table  18  lists  the  mean  satisfaction 
ratings  and  the  number  of  respondents  who  evaluated  them. 
The  rated  satisfaction  factors  are  shown  as  follows  in  the 
table: 

System  performance  =  PFTOT 
Additional  features  =  FETOT 
Serviceability  =  SVTOT 
Aesthetics  =  AETOT 
Computer  Satisfaction  =  COMPSAT. 

As  discussed  above,  the  satisfaction  rating  of  each  of 
the  above  first  four  factors  was  weighted  using  its  corre¬ 
sponding  importance  rating.  The  computer  satisfaction 
rating  was  calculated  from  a  mean  of  the  responses  to 
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question  60  from  the  user  survey  shown  in  Appendix  D.  The 
total  satisfaction  rating  was  calculated  by  taking  the  mean 
of  the  five  satisfaction  ratings.  The  satisfaction  ratings 
can  be  compared  against  the  following  scale  which  was  used 
in  the  user  survey: 

1 _ 2 _ 3 _ 4 _ 5 _ 6 

no  very  low  moderate  high  very 

opinion  low  high 


TABLE  18 

SATISFACTION  RATINGS  FOR  ALL  MEDICAL  CENTERS 


Computer  Support 


Mean  Rating 


PFTOT 

FETOT 

SVTOT 

AETOT 

COMPSAT 

TOTAL 


3.365 

2.877 

3.526 

2.811 

3.995 

3.315 


Medical  Center  Satisfaction  Ratings. 

As  described  above,  satisfaction  ratings  were  calculated  by 
medical  center  for  information  systems  which  were  in  opera¬ 
tion.  Table  19  shows  the  satisfaction  rating  of  the  infor¬ 
mation  systems  for  each  medical  center.  The  medical  cen¬ 
ters  are  ranked  in  descending  order  of  total  satisfaction 
rating.  The  names  of  the  medical  centers  are  abbreviated 
as  follows: 

David  Grant  Medical  Center  =  DGMC 
Keesler  Medical  Center  =  KMC 
Malcolm  Grow  Medical  Center  =  MGMC 
Scott  Medical  Center  =  SMC 
Wilford  Hall  Medical  Center  =  WHMC 
Wright-Patterson  Medical  Center  =  WPMC. 
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TABLE  19 


SATISFACTION  RATINGS  FOR  INDIVIDUAL  MEDICAL  CENTERS 


mi 

»iM »  Mil 

» w .  an 

0P**M 

ti  i’vi 

fc  M  •  V  . 

«*  1  *  V  * 

AL  »  w  . 

CCXPS&T 

i 

A 

SGMC 

3.413 

3.390 

2.914 

3.522 

3.329 

4.255 

A 

4 

me 

3.376 

3.401 

•  2.992 

3.519 

1  Art  i 

M  *  *  Vv 

3.966 

3 

2GMC 

3.344 

3.541 

2.325 

3.883 

2.315 

3.553 

4 

KMC 

3.277 

3.141 

2.919 

3.441 

2.742 

1  1  J7 

1  .  A*X4 

5 

me 

3.239 

3.157 

2.681 

?  CM 

■d  •  J 

1.544 

1  M’ 

7  .  v  Di. 

6 

SMC 

3.123 

3.158 

•>  a*n 

4  . 044 

7  Caa 

7  AM 

M  •  to  *  # 

*  77  * 

The  researchers  used  a  Pearson's  correlation  coeffi¬ 
cient  to  see  if  the  overall  computer  system  satisfaction 
rating  was  related  to  any  of  the  demographic  characteristics 
of  the  respondents.  This  test  showed  that  the  demographics 
of  respondents  (i.e.,  gender,  age,  educational  level,  duty 
status,  rank,  grade,  corps,  computer  experience,  computer 
training,  and  occupational  longevity)  were  independent  of 
the  satisfaction  rating  (i.e.,  none  of  the  measured  charac¬ 
teristics  of  respondents  led  to  differentiated  responses  to 
system  satisfaction) . 

Information  System  Satisfaction  Ratings.  Mean 
satisfaction  ratings  across  medical  centers  were  calculated 
for  individual  information  systems  that  were  evaluated  by  at 
least  20  respondents.  The  researchers  chose  this  cutoff  to 
show  those  systems  which  respondents  rated  the  most  fre¬ 
quently.  There  were  14  such  systems.  Table  20  lists  the 
systems  along  with  their  satisfaction  ratings  and  the  number 
of  respondents  who  evaluated  them.  Descriptions  of  the 
systems  shown  in  Table  20  are  found  in  Appendix  F. 
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The  rated  satisfaction  factors  are  abbreviated  as  follows: 

System  performance  =  PPTOT 
Additional  features  =  FETOT 
Serviceability  =  SVTOT 
Aesthetics  =  AETOT. 


TABLE  20 

SATISFACTION  RATINGS  BY  SYSTEM 


RANK  SYSTEM 

TOTAL 

PFTOT 

FETOT 

SVTOT 

AETOT 

n 

1 

♦Managed  H.C. 

3.327 

3.285 

3.017 

3.760 

3.248 

63 

2 

**CMIS 

3.264 

3.422 

3.233 

3.554 

2.846 

33 

3 

***lab 

3.206 

3.483 

3.000 

3.738 

2.647 

44 

4 

CHCS 

3.165 

3.282 

2.933 

3.563 

2.882 

159 

5 

Tripharm 

3.156 

3.310 

2.871 

3.632 

2.811 

22 

6 

LAN 

3.102 

3.189 

2.814 

3.583 

2.825 

77 

7 

EASIII/MEPRS 

3.082 

3.145 

2.405 

3.431 

3.077 

30 

8 

Wang  PAS 

3.030 

3.352 

2.600 

3.400 

2.767 

28 

9 

AQCESS 

2.995 

3.190 

2.587 

3.425 

2.776 

151 

■10 

Wang  Office 

2.960 

3.121 

2.636 

3.471 

2.610 

53 

11 

MEDL0G/MM4S 

?.893 

3.217 

2.581 

3.153 

2.621 

51 

12 

PCIII 

2.871 

2.808 

2.832 

3.409 

2.435 

24 

13 

Trilab 

2.742 

3.215 

2.248 

3.219 

2.285 

120 

14 

Trirad 

2.620 

2.848 

2.333 

3.055 

2.245 

26 

*  Wright-Pattersan  Medical  Center  only 
**  Keesler  Medical  Center  only 
***  Wilford  Hall  Medical  Center  only 
Note:  Descriptions  of  the  above  systems  are  found  in  Appendix  F. 


As  discussed  in  Chapter  III,  the  satisfaction  rating  in 
each  of  the  above  four  factors  was  weighted  using  its 
corresponding  importance  rating.  The  total  satisfaction 
rating  was  derived  by  calculating  the  mean  of  the  above 
satisfaction  factor  ratings.  The  systems  were  then  ranked 
by  total  satisfaction  rating.  The  three  highest  ranked 
systems  were  developed  and  implemented  by  individual  medical 
ceftters  and  therefore  are  unique  to  that  location. 
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In  Table  20,  the  satisfaction  ratings  can  be  directly 
compared  to  the  scale  in  the  User  survey  shown  at 
Appendix  D.  If  the  average  worker  feels  at  all  comfortable 
using  a  system,  the  satisfaction  rating  for  the  system 
should  range  from  at  least  a  3.000  to  a  4.000  which  corre¬ 
sponds  to  a  low  to  moderate  satisfaction  rating. 

Comments .  The  users  were  encouraged  to  provide 
comments  on  the  User  survey  in  question  62  as  shown  at 
Appendix  D.  Of  the  718  useable  responses,  92  respondents 
took  the  time  to  provide  comments  about  the  quality  of 
information  system  support  in  their  medical  center.  There 
were  sufficient  comments  made  about  information  system 
training  and  CHCS  to  provide  a  summary  of  those  responses. 

One  strong  theme  emerged  from  the  comments  of  respon¬ 
dents  pertaining  to  training.  Many  respondents  were  con¬ 
cerned  that  they  had  not  received  adequate  training  to  use 
the  information  systems  present  at  their  work  site.  They 
felt  that  proper  training  on  how  to  use  the  systems  would 
improve  their  productivity  at  work.  This  theme  seems  to 
reinforce  the  information  provided  in  Table  14  about  the 
level  of  computer  training  received  by  the  system  users. 
Table  14  revealed  that  33.2  percent  of  the  respondents  had 
taught  themselves  to  operate  the  information  systems  while 
7.7  percent  had  not  received  any  computer  training. 
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At  Keesler  and  Malcolm  Grow  Medical  Centers,  there  were 
seven  comments  made  at  each  site  about  the  new  medical 
system,  CHCS.  Opinions  were  mixed  about  the  reliability  and 
the  usefulness  of  CHCS.  Users  who  liked  the  system  said 
that  it  was  wonderful  and  very  user  friendly  compared  to 
other  medical  information  systems.  As  one  would  expect  for 
a  newly  implemented  system,  many  users  expressed  frustration 
with  what  they  perceived  as  flaws  in  the  system.  One  common 
complaint  was  that  the  system  was  cumbersome  to  use  and  not 
user  friendly.  Another  frequent  complaint  was  that  patient 
appointments  "mysteriously"  disappeared  from  the  appointment 
system  after  they  had  been  input. 

System  Summaries 

In  Phase  Three  of  the  research,  pertinent  data  was 
gathered  about  the  most  widely  used  Air  Force  medical 
information  systems.  As  discussed  in  Chapter  III,  this 
information  was  needed  to  give  the  researchers  further 
insight  into  the  interpretation  of  the  results  from  the  CIO 
and  user  surveys.  The  researchers  and  sponsor  determined 
that  such  individual  system  factors  as  age,  maintenance  and 
manpower  requirements,  and  quantity,  quality,  and  type  of 
information  required  could  potentially  effect  the  level  of 
service  quality  provided  by  the  medical  center's  MSO. 

Descriptions  of  those  systems  that  were  evaluated  by  at 
least  20  user  survey  respondents,  as  shown  in  Table  20,  are 
found  at  Appendix  F. 
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Summary  of  Analysis 

This  chapter  presented  and  discussed  the  findings  from 
the  analysis  of  the  data  gathered  in  the  last  three  phases 
of  the  research.  The  findings  in  this  chapter  answered 
Investigative  Questions  1,  4,  and  5. 

First,  the  responses  to  the  CIO  survey  were  summarized. 
The  response  rate  for  this  survey  was  100  percent.  The  MSO 
demographics  including  personnel  functions  and  personnel 
staffing  were  presented  in  Tables  6  and  7.  Table  8  pre¬ 
sented  a  roster  of  information  systems  found  at  each  medical 
center.  Finally,  the  control  statistics  reported  by  each 
medical  center  were  discussed. 

Next,  the  responses  to  the  system  user  survey  were 
analyzed  and  summarized.  The  response  rate  for  this  survey 
was  42.5  percent.  The  results  of  a  respondent  representa¬ 
tiveness  assessment  from  the  survey  responses  were 
presented.  From  this  assessment,  the  researchers  cannot 
claim  that  the  system  users  are  uniformly  represented  among 
medical  centers  but  they  believe  the  response  rate  was 
adequate  from  each  medical  center  to  draw  some  conclusions 
from  the  collected  data.  A  summary  of  respondent 
demographics  was  presented  which  included  personal  data 
along  with  level  of  experience,  training,  and  confidence  in 
using  computers.  Information  from  the  analysis  of  the  user 
survey  was  used  to  develop  total  satisfaction  ratings. 

First,  total  information  system  satisfaction  ratings  were 


111 


presented  by  information  system  without  regard  to  medical 
center.  Next,  satisfaction  ratings  were  presented  collect¬ 
ively  for  each  information  system  by  medical  center  for 
which  sufficient  data  were  available.  Finally,  individual 
information  system  ratings  were  presented  for  each  informa¬ 
tion  system  regardless  of  medical  center  in  which  it  is 
operational.  User  comments  included  in  the  survey  responses 
were  summarized.  The  users  were  concerned  about  the  lack  of 
training  provided  on  the  information  systems.  A  summary  of 
comments  was  also  provided  about  CHCS. 

Descriptions  of  the  information  systems  listed  in  Table 
20  which  were  evaluated  by  at  least  20  survey  respondents, 
were  provided  in  Appendix  F.  The  descriptions  included  such 
individual  system  factors  as  age,  maintenance,  and  manpower 
requirements  which  could  potentially  effect  the  level  of 
service  quality  provided  by  the  medical  center's  MSO. 

This  chapter  described  the  analysis  performed  on 
responses  to  the  survey  instruments  sent  to  the  CIOs  and 
information  system  users  in  the  CONUS-based  USAF  medical 
centers.  Chapter  V  will  continue  with  a  discussion  of  the 
findings  in  this  chapter  followed  by  recommendations  for 
future  research  and  an  overall  conclusion. 
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V.  Findings.  Recommendations,  and  Conclusion 

Introduction 

This  thesis  has  considered  the  question  of  whether 
there  is  a  relationship  between  the  source  of  MSO  staffing 
and  the  level  of  quality  of  the  services  provided  by  medical 
systems  offices  in  Air  Force  medical  centers.  In  so  doing 
the  researchers  have  sought  to  determine  that  MSO  staffing 
method  whose  information  services  are  of  the  highest  rela¬ 
tive  quality. 

Chapter  I  introduced  general  issues  surrounding  the 
current  information  service  staffing  environment  in  Air 
Force  medical  centers.  Air  Force  Program  Action  Directive 
90-4  was  initially  proposed  because  Air  Force  information 
experts  believe  that  computer  system  management  is  best 
provided  by  communications/computer  specialists.  However, 
some  Air  Force  medical  center  CIOs  have  questioned  the 
ability  of  nonmedical  computer  specialists  to  cater  to  the 
information  needs  of  the  military  medical  community. 

A  literature  review  was  performed  in  Chapter  II  to 
better  understand  current  trends  in  medical  information 
services  and  the  measurement  of  system  quality.  Juran 
(1989)  and  Ernst  and  Young  (1990)  have  developed  models  for 
measuring  product  or  service  quality,  both  of  which  empha¬ 
size  the  importance  of  conforming  to  the  standards  of  the 
industry  and  meeting  the  requirements  of  the  customers. 
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When  a  producer’s  goods  and/or  services  conform  with  indus¬ 
try  standards,  a  neutral  level  of  quality  is  attained. 
However,  not  until  the  customer  is  satisfied  with  the  good 
or  service  can  it  be  considered  to  be  of  high  quality  (Ernst 
and  Young, 1990). 

Garvin  (1988)  enhanced  the  model  by  identifying  eight 
dimensions  of  quality  as  precise  measurement  criteria. 

These  dimensions  are:  performance,  features,  reliability, 
conformance,  durability,  serviceability,  aesthetics,  and 
perceived  quality  (Garvin, 1988) .  The  researchers  determined 
that  all  of  the  quality  dimensions  except  durability  could 
be  measured  in  this  study.  This  determination  was  made 
because  the  ultimate  product  of  an  information  system  is  the 
information  it  provides.  The  hardware  and  software  that 
produce  the  information  will  periodically  be  replaced  and 
updated,  but  the  lifespan  of  the  information  itself  is 
dictated  solely  by  external  factors  (Ahituv  and  Neumann, 
1990) . 

Further,  several  personal  characteristics  were  identi¬ 
fied  as  being  necessary  among  managers  of  information  sys¬ 
tems.  Those  characteristics  are  technological  competence, 
highly  developed  client  relations  skills,  knowledge  of  the 
organization's  business  and  mission,  and  loyalty  to  the 
organization.  This  broad  range  of  necessary  skills  has 
sparked  much  debate  in  the  literature  as  to  whom  should 
manage  the  organization's  information  skills:  either 
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in-house  personnel  who  are  knowledgeable  about  the  organiza¬ 
tion's  industry  and  are  loyal  to  the  organization,  or  people 
from  enterprises  whose  trade  is  providing  information  serv¬ 
ices  and  who  might  be  considered  to  be  more  technically 
competent.  In  the  literature,  neither  group  demonstrated  a 
higher  level  of  client  relations  skills. 

Finally,  Chapter  I  discussed  the  standards  and  goals  of 
the  Air  Force  Medical  Service  for  providing  information 
services  and  the  responsibilities  of  the  CIOs  at  the  indi¬ 
vidual  Air  Force  medical  treatment  facilities.  All  six  USAF 
medical  centers  are  successfully  meeting  those  standards  and 
goals  according  to  the  Chief  of  the  Air  Force  Medical  Serv¬ 
ice  Information  Systems  Division  (Obuchowski,1992) .  This  is 
a  first  step  toward  meeting  the  conformance  criterion  of 
quality. 

Two  survey  instruments,  one  for  USAF  medical  center 
CIOs  and  another  for  information  system  users,  were 
developed  in  order  to  assess  medical  center  MSO  staffing  and 
the  quality  of  services  provided  by  the  MSOs .  These  surveys 
are  found  at  Appendices  C  and  D. 

The  purpose  of  the  CIO  survey  was  determine  the  source 
of  each  individual  MSO  staff  member,  the  specific  informa¬ 
tion  systems  available  at  each  medical  center,  and  the 
system  control  functions  performed  at  each  site.  Surveys 
were  sent  to  the  CIOs  at  all  six  USAF  medical  centers  in  the 
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continental  United  States.  The  response  rate  was  100  per¬ 
cent  . 

The  objective  of  the  user  survey  was  to  assess  the 
quality  of  information  services  at  the  medical  centers  on 
the  basis  of  performance,  features,  serviceability,  aesthet¬ 
ics,  and  overall  perceived  quality.  The  respondents  were 
given  the  opportunity  to  evaluate  up  to  three  different 
information  systems.  The  surveys  were  sent  to  1690  system 
users  at  the  six  sites,  718  of  which  were  returned  to 
researchers  with  usable  data  for  a  response  rate  of  42.5 
percent.  An  extensive  analysis  of  the  responses  to  both 
surveys  was  performed,  the  results  of  which  were  reported  in 
Chapter  IV. 

This  chapter  discusses  the  pertinent  information  from 
the  surveys  organized  by  medical  center,  specifically 
addressing  the  quality  issues  as  they  relate  to  the  method 
of  staffing  at  each  site.  Further,  the  researchers  will 
make  recommendations  for  action  and  for  further  research  and 
draw  an  overall  conclusion  regarding  MSO  staffing  and 
quality  in  this  chapter. 

Findings 

The  user  surveys  sought  to  determine  the  level  of 
service  quality  provided  by  information  systems  in  USAF 
medical  centers.  Quality  was  measured  in  the  survey  on  five 
dimensions:  performance,  additional  features,  service¬ 

ability,  aesthetics,  and  overall  computer  system 
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satisfaction.  Users'  satisfaction  with  each  of  these  dimen¬ 
sions  was  measured  using  a  Likert  continuous  scale  as  fol- 
1  ows : 

1 - 2 - 3 - 4 - 5 - 6 

no  very  low  moderate  high  very 

opinion  low  high 

Answers  of  "1”  or  "no  opinion”  were  considered  to  be  nonre¬ 
sponses  and  were  not  calculated  into  the  mean  ratings, 
therefore  each  response  had  a  5-point  range  with  a  median  of 
"4"  or  "moderate."  The  five  dimension  scores  were  averaged 
together  resulting  in  a  total  quality  rating. 

The  total  quality  rating  across  all  six  medical  centers 
was  3.315,  which  on  the  Likert  scale  is  in  the  low  to  moder¬ 
ate  range. 

Overall  satisfaction  with  medical  center  computer 
systems  received  the  highest  individual  dimension  score  with 
a  mean  of  3.995.  Specifically,  the  respondents  were  asked 
rate  their  level  of  satisfaction  with  the  medical  center 
computer  system's  ability  to  help  them  perform  their  jobs. 

Serviceability  was  given  a  mean  rating  of  3.526  by  the 
users.  Survey  respondents  were  provided  with  the  following 
definition  of  serviceability:  "the  amount  of  time  the  com¬ 
puter  system  is  in  full  service  to  the  users  and  the  degree 
to  which  the  medical  information  system  specialists  provide 
prompt,  knowledgeable,  and  courteous  maintenance." 

Satisfaction  with  performance  received  the  next  highest 
mean  score  of  3.365.  In  the  survey,  performance  was  defined 
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as  a  measure  of:  "how  well  the  user  thinks  the  computer 
system  does  what  he  or  she  thinks  it  should  do,  such  as 
readability  and  usefulness  of  reports  and  printouts, 
availability  of  necessary  information,  and  correctness  of 
output . " 

The  mean  score  for  users'  satisfaction  with  the  com¬ 
puter  systems'  additional  features  was  2.877.  Additional 
features  were  defined  as:  "those  options  that  add  to  the 
computer  system's  functionality,  but  are  not  required  in 
performing  duties  at  the  medical  center."  Examples  of  addi¬ 
tional  features  include  menus,  on-line  help,  and  virus 
scanning . 

Finally,  aesthetics  received  the  lowest  mean  score  of 
2.811.  For  the  purposes  of  this  study,  aesthetics  are  a 
measure  of:  "the  degree  the  computer  system  appeals  to  the 
user's  senses  in  such  things  as  monitor  screen  colors, 
appearance  of  reports  and  printouts,  location  of  computer 
terminals,  and  operator  comfort  at  computer  work  stations." 

Because  survey  respondents  were  given  the  opportunity 
to  assess  their  satisfaction  with  up  to  three  individual 
computer  systems  at  their  medical  facility,  the  researchers 
were  able  to  calculate  mean  satisfaction  ratings  for  many  of 
the  systems  available  at  the  medical  centers.  Mean  ratings 
were  only  calculated  for  those  systems  evaluated  by  at  least 
20  survey  respondents.  This  number  was  chosen  because  the 
researchers  felt  evaluations  by  fewer  than  20  respondents 
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would  not  be  sufficient  to  make  assumptions  about  the  entire 
population. 

The  individual  systems  were  evaluated  on  four  quality 
dimensions:  performance,  additional  features,  service¬ 

ability,  and  aesthetics.  The  mean  scores  for  each  of  these 
dimensions  were  averaged  together  to  arrive  at  a  total 
score.  Table  20  in  Chapter  IV  shows  the  14  systems  for 
which  these  scores  were  calculated  and  the  relative  system 
rankings  by  total  score. 

The  three  highest  total  ratings  were  scored  by  systems 
that  were  developed  and  implemented  by  a  single  medical 
center,  specifically  Managed  Health  Care  at  Wright-Patterson 
(3.327),  CMIS  at  Keesler  (3.264),  and  LAB  at  Wilford  Hall 
(3.206). 

The  Composite  Health  Care  System  (CHCS),  a  system 
developed  by  the  Department  of  Defense  and  currently  in  its 
final  beta  testing  stages  at  DOD  medical  facilities,  was 
ranked  number  four  with  a  total  rating  of  3.165.  At 
present,  it  is  only  available  at  two  Air  Force  medical 
centers,  Malcolm  Grow  and  Keesler. 

The  individual  system  ratings  will  be  discussed  in  more 
detail  in  the  following  sections  as  they  relate  to  each  of 
the  medical  centers'  total  scores. 

Individual  Medical  Center  Ratings.  Table  21  is  a 
consolidation  of  the  data  from  Tables  7,  8,  19,  and  20  from 
Chapter  IV.  The  second  column  lists  the  rankings  of  each 
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medical  center  relative  to  the  others  in  system  user  satis¬ 
faction.  The  fourth  column  shows  the  rank  of  each 
information  system's  user  satisfaction  rating  across  medical 
centers . 


TABLE  21 

MEDICAL  CENTERS,  SYSTEMS,  AND  STAFFING 


Medical  Canter 

Rank 

Rated  Systems 

System  Ranks 

Staffing  Sources 

Supervision 

Malcolm  Crow 

1 

CHCS 

4 

Mil  Med  (12.3%) 

Med  Ctr  (81.3%) 

ACCESS 

9 

Mil  SC  (25  %) 

Outside  (13.7%) 

Trilab 

13 

Civ  Govt (43. 8%) 

Tripharm 

5 

Contract (18. 7%) 

Trirad 

14 

Wilford  Hall 

2 

LAB 

3 

Mil  Med  (9.3%) 

Med  Ctr  (74.5%) 

ACCESS 

9 

Mil  SC  (7.8%) 

Outside  (25.5%) 

EASI1I/MEPRS 

7 

Civ  Govt (56.9%) 

MEDLOG 

11 

Contract (25. 5%) 

PCI1I 

12 

Tripharm 

5 

Trirad 

14 

David  Crant 

3 

ACCESS 

9 

Mil  Med  (SO  %) 

Med  Ctr  (100  %) 

EASIII/MEPRS 

7 

MEDLOG 

11 

Civ  Govt (50  %) 

Tripharm 

5 

Trirad 

14 

Keesler 

4 

CMIS 

2 

Mil  Med  (13.3%) 

Med  Ctr  (43.3%) 

ACCESS 

9 

Mil  SC  (10  %) 

Outside  (56.7%) 

CHCS 

4 

Civ  Govt (20  %) 

PCIII 

12 

Contract (56. 7%) 

Tripharm 

5 

Wright -Patter son 

i  3 

Managed  HC 

1 

Mil  Med  (28.1%) 

Med  Ctr  (56.2%) 

ACCESS 

9 

Mil  SC  (46.9%) 

Outside  (43.8%) 

EASIII/MEPRS 

7 

Civ  Govt (18. 8%) 

Trilah 

13 

Contract  (6.2%) 

Tripharm 

5 

Scott 

6 

ACCESS 

9 

Mil  Med  (14.3%) 

Med  Ctr  (100  %) 

EASIII/MEPRS 

7 

MEDLOG 

11 

Civ  Govt(85.7%) 

Trilab 

13 

Tripharm 

3 

Trirad 

14 

120 


Andrews  Air  Force  Base.  Surveys  were  distributed 
to  230  users  at  Andrews'  Malcolm  Grow  USAF  Medical  Center  of 
which  149  were  returned  to  the  researchers  in  usable  form 
for  a  64.8  percent  response  rate.  Malcolm  Grow  achieved  the 
highest  total  satisfaction  rating  of  any  other  medical 
center  with  a  score  of  3.418.  They  also  received  the  high¬ 
est  ratings  in  the  computer  system  satisfaction  and  aesthet¬ 
ics  dimensions. 

Of  the  14  computer  systems  given  satisfaction  ratings, 
Malcolm  Grow  has  five  in  operation:  AQCESS  (ranked  number  9 
across  medical  centers),  CHCS  (ranked  number  4),  Trilab 
(ranked  number  13),  Tripharm  (ranked  number  5),  and  Trirad 
(ranked  number  14). 

The  MSO  at  Malcolm  Grow  is  staffed  with  16  people  and 
is  made  up  of  staff  members  from  each  of  the  four  major 
source  categories,  namely  military  medical,  military  SC, 
civilian  government,  and  contractor.  The  largest  portion  is 
civilian  government  at  almost  44  percent.  The  MSO  staff 
member  to  total  center  staff  ratio  is  1  to  100. 

Lackland  Air  Force  Base.  A  total  of  450  surveys 
were  distributed  to  system  users  at  Wilford  Hall  USAF  Medi¬ 
cal  Center  at  Lackland  Air  Force  Base.  Of  those,  135  were 
returned  to  the  researchers  in  usable  form  for  a  30  percent 
response  rate.  Wilford  Hall  was  second  only  to  Malcolm  Grow 
in  its  total  satisfaction  rating  of  3.376.  Additional 
features  were  rated  highest  of  any  of  the  medical  centers. 
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Wilford  Hall  has  7  of  the  14  evaluated  systems  in 
operation:  LAB  (ranked  number  3  across  medical  centers), 

AQCESS  (ranked  number  9),  EASIII/MEPRS  (ranked  number  7), 
MEDLOG  (ranked  number  11),  PCI  1 1  (ranked  number  12),  Tri- 
pharm  (ranked  number  5),  and  Trirad  (ranked  number  14), 

Wilford  Hall's  MSO  is  staffed  by  51  people,  42  of  whom 
are  civilians.  Almost  60  percent  of  the  staff  are  civilian 
government  employees  while  about  25  percent  are  contracted 
personnel.  The  other  two  source  categories  are  respresented 
on  the  staff  as  well.  About  75  percent  of  the  staff  are 
supervised  by  people  within  the  medical  center.  There  is 
one  MSO  staff  member  for  every  88  people  on  the  medical 
center  staff. 

Travis  Air  Force  Base.  Two-hundred  thirty  users 
at  David  Grant  USAF  Medical  Center  were  given  surveys,  94  of 
whom  returned  them  in  usable  form.  This  number  represented 
a  response  rate  of  40.9  percent.  David  Grant's  user  satis¬ 
faction  rating  ranked  third  among  USAF  medical  centers.  The 
medical  center  was  ranked  at  the  top  in  two  of  the  quality 
dimensions:  performance  and  serviceability. 

David  Grant  has  5  of  the  evaluated  systems  in  service 
locally:  AQCESS  (ranked  number  9  across  medical  centers), 

EASIII/MEPRS  (ranked  number  7),  MEDLOG  (ranked  number  11), 
Tripharm  (ranked  number  5),  and  Trirad  (ranked  number  14). 

There  are  12  people  on  Travis'  MSO  staff  equally 
divided  between  military  medical  and  civilian  government. 
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All  of  the  staff  members  are  supervised  by  people  within  the 
medical  center.  David  Grant's  MSO  staff  to  medical  center 


staff  ratio  is  about  one  to  133. 

Keesler  Air  Force  Base.  Keesler  USAF  Medical 
Center  had  a  38.3  percent  user  survey  response  rate  with  111 
returned  usable  responses  of  the  290  distributed.  Keesler's 
total  user  satisfaction  score  was  3.277,  ranking  fourth 
among  the  medical  centers.  The  medical  center  ranked  second 
in  overall  computer  system  satisfaction  and  additional 
features . 

Five  of  the  14  evaluated  systems  are  operational  at 
Keesler:  CMIS  (ranked  number  2  across  medical  centers), 

AQCESS  (ranked  number  9),  CHCS  (ranked  number  4),  PCI  1 1 
(ranked  number  12),  and  Tripharm  (ranked  number  5). 

The  MSO  staff  at  Keesler  is  equally  divided  between 
military  medical  and  civilian  government.  All  of  the  staff 
members  are  supervised  by  people  within  the  medical  center. 
There  is  one  MSO  staff  member  for  about  every  66  people  on 
the  medical  center  staff. 

Wright-Patterson  Air  Force  Base.  One  hundred 
three  user  surveys  from  Wright-Patterson  were  returned  to 
the  researchers  of  the  260  distributed  for  an  overall 
response  rate  of  39.6  percent.  The  respondents  gave  the 
medical  center  a  total  user  satisfaction  score  of  3.209, 
fifth  ranked  among  the  medical  centers. 
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Wright-Patterson ' s  in-house  developed  and  implemented 
information  system,  Managed  Health  Care,  was  ranked  number 
one  among  the  14  systems  evaluated  in  total  user  satisfac¬ 
tion  with  a  score  of  3.327.  The  system  also  scored  the 
highest  ratings  in  serviceability  and  aesthetics.  Also  in 
operation  at  Wright-Patterson  are  AQCESS  (ranked  number  9 
across  medical  centers),  EASIII/MEPRS  (ranked  number  7), 
Trilab  (ranked  number  13),  and  Tripharm  (ranked  number  5). 

There  are  32  people  on  the  Wright-Patterson  MSO  staff. 
This  is  the  only  Air  Force  medical  center  that  has  SC  super¬ 
vised  military  computer  specialists  on  its  staff.  There  are 
also  3  military  SC  staff  members  who  are  supervised  by 
medical  center  personnel.  The  military  medical,  civilian 
government,  and  contractor  staff  source  categories  are  also 
represented.  The  MSO  staff  to  medical  center  staff  ratio  at 
Wright-Patterson  is  1  to  56,  the  lowest  among  the  medical 
centers . 

Scott  Air  Force  Base.  User  surveys  were  distrib¬ 
uted  to  230  computer  system  customers  at  Scott  USAF  Medical 
Center  of  which  124  were  returned  (53.9  percent).  The  total 
user  satisfaction  score  at  Scott  was  3.158.  The  aesthetics 
satisfaction  portion  of  the  rating  was  highest  among  the 
medical  centers.  Serviceability  was  second  highest. 

Six  of  the  evaluated  information  systems  are  in  opera¬ 
tion  at  Scott:  AQCESS  (ranked  number  9),  EASIII/MEPRS 
(ranked  number  7),  MEDLOG  (ranked  number  11),  Trilab  (ranked 
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number  13),  Tripharm  (ranked  number  5),  and  Trirad  (ranked 
number  14) . 

Scott  has  seven  people  on  the  MSO  staff,  six  of  whom 
are  civilian  government  employees.  The  seventh  person  is 
military  medical.  All  seven  staff  members  are  supervised 
from  within  the  medical  center.  There  is  one  MSO  staff 
member  for  every  143  medical  center  employees,  the  highest 
ratio  among  the  medical  centers. 

Summary  of  Findings .  The  difference  between  the  high¬ 
est  total  satisfaction  rating  at  Malcolm  Grow  (3.418)  and 
the  lowest  rating  at  Scott  (3.158)  is  less  than  0.3.  Fur¬ 
ther,  the  lowest  rating  is  only  about  0.16  lower  than  the 
total  satisfaction  rating  across  medical  centers  (3.315). 
This  narrow  range  of  scores  between  the  highest  and  lowest 
led  the  researchers  to  conclude  that  the  difference  in  the 
level  of  information  service  quality  across  medical  centers 
is  too  small  to  be  considered  significant  at  a  =  0.05. 
However,  the  difference  is  significant  at  a  =  0.1  (z  =  1.6; 
0 . 05<p<0 . 10) . 

Even  if  the  differences  among  medical  centers  were  to 
be  considered  significant  (at  p=0.10),  differences  in  the 
level  of  service  quality  provided  to  customers  at  the  medi¬ 
cal  centers  is  not  due  to  the  sources  of  MSO  staffing  or 
upervision.  No  noticeable  trends  in  methods  of  staffing 
appeared  from  the  data  and  every  MSO,  to  some  extent,  was 
staffed  by  people  belonging  to  more  than  one  source 
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category.  Further,  MSO  staff  to  medical  center  staff  ratio 
does  not  appear  to  be  a  factor  in  determinine  quality  of 
information  services.  The  MSO  with  the  lowest  ratio  was 
ranked  fifth  in  overall  user  satisfaction  while  the  MSO  with 
the  highest  ratio  was  ranked  sixth.  The  medical  center  with 
the  most  satisfied  information  system  users  has  an  MSO 
staffing  ratio  that  is  halfway  between  the  lowest  and  high¬ 
est  ratios. 

Moreover,  the  operational  information  systems  at  each 
medical  center  seemed  to  play  little  part  in  the  medical 
center's  total  satisfaction  rating.  All  six  sites  have  at 
least  one  system  ranked  in  the  top  five  on  the  basis  of  user 
satisfaction  and  at  least  one  in  the  bottom  five. 

* 

Recommendations 

The  researchers  have  divided  their  recommendations  from 
this  study  into  two  types.  The  first  involves  recommenda¬ 
tions  for  action  by  the  Air  Force  Medical  Service  and  the 
individual  medical  center  MSOs .  The  second  group  of  recom¬ 
mendations  is  suggested  areas  for  future  research. 

Recommendations  for  Action.  Because  staffing  method 
and  source  of  supervision  appeared  to  have  no  influence  on 
the  quality  of  service  provided  by  MSOs,  the  researchers 
recommend  that  the  individual  medical  centers  continue  with 
the  practice  of  staffing  their  MSOs  to  meet  their  own 
requirements  and  constraints.  Management  of  information 
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systems  in  Air  Force  medical  centers  should  not  be  centrally 
standardized . 

Although  no  conclusions  could  be  drawn  relating  infor¬ 
mation  system  user  satisfaction  to  MSO  staffing  sources, 
other  trends  did  become  evident  during  the  course  of  this 
study.  One  of  those  trends  is  that  USAF  medical  center 
information  system  users  seem  to  be  more  satisfied  with 
those  systems  that  were  developed  and  implemented  at  a 
single  medical  center  than  with  those  developed  by  DOD  or 
Air  Force  agencies  for  implementation  at  multiple  locations. 

There  are  several  possible  explanations  fur  this  trend. 
One  explanation  might  be  that  local  developers  are  in  closer 
contact  with  customers  and  are,  therefore,  more  able  to 
tailor  the  system  to  meet  the  unique  requirements  at  that 
site.  Customers  are  able  to  voice  their  opinions  and  con¬ 
cerns  about  the  features  of  the  system.  Further,  the  cus¬ 
tomers  feel  more  of  a  sense  of  ownership  for  the  system  once 
it  is  operational.  Another  explanation  might  be  that 
developers  at  a  single  site  do  not  need  to  deal  with  con¬ 
straints  of  standardization.  They  only  need  to  consider 
interface  with  hardware,  software,  and  operating  systems 
already  in  place  at  their  own  location  rather  than  those  of 
many  locations. 

Regardless  of  the  explanation  for  this  phenomenon,  the 
researchers  recommend  that  the  Air  Force  Medical  Service 
continue  to  encourage  these  innovative  projects  at 
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individual  medical  treatment  facilities.  In  addition,  AFMSA 
should  widely  disseminate  information  about  these  single 
location  projects  among  all  Air  Force  hospitals  and  clinics. 
Many  of  the  projects  could  potentially  be  implemented  at 
other  sites  with  minimal  adaptation. 

A  second  trend  found  by  the  researchers  was  that  system 
users  are  least  satisfied  with  the  aesthetics  and  additional 
features  dimensions.  It  is  recommended  that  not  only  the 
staff  at  AFMSA,  but  also  the  CIOs  at  the  medical  centers, 
explore  ways  to  improve  Air  Force  medical  information  sys¬ 
tems  in  these  dimensions.  Using  the  definitions  from  the 
user  survey,  these  dimensions  are  those  that  make  the  user's 
job  at  the  computer  a  little  easier  and  more  pleasant.  The 
researchers  suggest  that  improvements  in  these  areas  may 
improve  productivity  among  hospital  staff  members  with  such 
things  as  enhanced  screen  color  capabilities,  more  comfort¬ 
able  work  stations,  help  menus,  and  on-line  tutorials. 

If  the  widespread  implementation  of  CHCS  continues  as 
scheduled  once  beta  testing  has  been  completed,  many  of  the 
systems  with  lower  total  satisfaction  ratings  will  be 
replaced.  Some  of  the  systems  whose  functions  will  be  taken 
over  by  CHCS  include  Automated  Quality  of  Care  Evaluation 
Support  System  (AQCESS),  Tri-Service  Laboratory  System 
(Trilab),  Tri-Service  Pharmacy  System  (Tripharm) ,  and  Tri- 
Service  Radiology  System  (Trirad).  Additionally,  as  CHCS  is 
developed  further,  on-line  interfaces  are  planned  with 
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Medical  Logistics  System  (MEDLOG)  and  Medical  Expense  and 
Performance  Reporting  System  (MEPRS)  (Air  Force  Medical 
Information  Systems  Plan, 1989). 

It  should  be  noted,  however,  that  CHCS  is  an  experimen¬ 
tal  system  under  DOD-wide  development,  and  should  be 
expected  to  have  some  early  implementation  difficulties, 
despite  which  it  ranked  fourth  among  evaluated  systems.  It 
may  be  that,  with  modifications  that  are  usual  with  software 
system  development,  this  system  may  prove  to  be  flexible  and 
equally  or  more  satisfactory  as  those  anomalies  and  defi¬ 
ciencies  are  corrected.  Further,  because  CHCS  received 
relatively  high  satisfaction  ratings  from  users,  the 
researchers  believe  that  overall  user  satisfaction  will  be 
improved  by  the  elimination  of  lower-rated  systems.  There¬ 
fore,  it  is  recommended  that  the  Air  Force  Medical  Service 
continue  with  its  plans  to  build  more  comprehensive  medical 
information  systems  with  more  initiatives  for  additional 
functionality  and  interface  with  other  stand-alone  systems. 

Many  of  those  users  who  took  the  time  to  provide  addi¬ 
tional  comments  in  the  section  provided  at  the  end  of  the 
user  survey  noted  the  lack  of  training  given  them  in  the  use 
of  medical  center  information  systems.  These  comments  are 
borne  out  in  Table  13,  Highest  Level  of  Computer  Training, 
in  Chapter  III.  The  researchers  recommend  that  more  time  be 
devoted  to  training  users  both  when  a  system  is  newly  imple¬ 
mented  and  on  a  continuing  basis. 
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Finally,  the  researchers  could  find  little  evidence  cf 
a  set  of  standards  for  measuring  the  technical  performance 
and  reliability  of  medical  information  systems  in  the  Air 
Force.  The  researchers  recommend  that  specific  guidelines 
be  developed  for  use  by  Air  Force  MSOs  to  monitor  such 
things  as  down  time,  mean  time  between  failures,  mean  time 
to  repair,  and  stress  work  loads.  Perhaps  these  guidelines 
could  be  developed  as  a  joint  effort  between  DOD  and  Head¬ 
quarters  Air  Force  level  system  experts  and  CIOs  at  the 
medical  treatment  facility  level.  These  guidelines  would 
enable  system  managers  to  more  precisely  measure  information 
system  quality  in  the  conformance  dimension  (Garvin, 1988 ) . 

TABLE  22 

RECOMMENDATIONS  FOR  ACTION 

_ | _ Recommended  Action _ 

1  Tailor  MSO  staffs  to  medical  center  needs 

2  Encourage  IS  development  at  medical  centers 

3  Improve  IS  aesthetics  and  additional  features 

4  Develop  comprehensive  ISs  with  more  function¬ 
ality  and  interface  with  stand-alone  systems 

5  Provide  more  user  training  throughout  the 
system  life  cycle 

6  Develop  guidelines  for  monitoring  system 

_ technical  performance  and  reliability _ 

Recommendations  for  Further  Research.  This  study 
revealed  several  opportunities  for  additional  research  in 
the  area  of  Air  Force  medical  information  system  quality. 
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This  research  was  conducted  using  only  Air  Force  medical 
centers.  Some  valuable  insight  might  be  gained  from  expand¬ 
ing  the  study  to  include  smaller  medical  treatment  facili¬ 
ties.  The  researchers  speculate  that  fewer  resources  are, 
in  general,  expended  in  the  smaller  hospitals  and  clinics 
toward  the  development  and  implementation  of  information 
systems.  It  would  be  of  use  to  system  planners  to  see  if 
users  at  these  smaller  facilities  are  more  or  less  satisfied 
with  available  information  systems  and  services  than  those 
at  the  medical  centers. 

The  second  topic  for  further  research  is  the  level  of 
user  satisfaction  after  full  implementation  of  CHCS.  Has 
the  level  of  user  satisfaction  improved  with  the  implementa¬ 
tion  of  CHCS  Air  Force  wide?  Future  researchers  might 
consider  comparing  the  data  from  this  project  gathered  from 
those  facilities  where  CHCS  is  not  yet  operational  to  data 
gathered  after  implementation  at  those  same  facilities. 

Another  opportunity  for  further  research  is  in  the  area 
of  users'  perceived  competence  in  using  specific  systems. 

In  this  study,  the  users  were  asked  to  evaluate  their  level 
of  competence  in  various  applications  such  as  word  process¬ 
ing,  spreadsheets,  and  electronic  mail.  Future  researchers 
could  determine  if  there  is  a  relationship  between  the 
users'  perceived  competence  and  perceived  quality  ratings 
for  individual  systems. 
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Final  1.  ';he  researchers  recommend  that  a  study  be 
undertaken  to  find  out  specifically  what  improvements  could 
be  made  to  present  information  systems  to  better  meet  the 
needs  of  users.  A  study  of  this  type  would  involve  a  survey 
of  users  of  specific  systems  across  medical  facilities  and 
at  every  level:  administrative  and  clinical,  technical  and 
managerial . 

TABLE  23 

RECOMMENDATIONS  FOR  FURTHER  RESEARCH 
_ | _ Further  Research  Recommendation _ 

1  Conduct  same  study  across  MTFs  of  all  sizes 

2  Conduct  same  study  after  full  impl ementaticn 
of  CHCS 

3  Conduct  study  comparing  user  perceived 
competence  with  system  satisfaction 

4  Conduct  study  of  users  to  determine  system 

_ improvements  to  better  meet  users'  needs _ 

Conclusion 

The  objective  of  this  research  was  to  determine  the 
extent  of  the  relationship  between  the  source  of  MSO  staff¬ 
ing  and  the  level  of  quality  of  the  services  provided  by 
medical  systems  offices  in  United  States  Air  Force  medical 
centers.  The  data  collected  during  the  course  of  this  study 
does  not  clearly  indicate  any  relationship  between  the 
source  of  MSO  staffing  and  the  level  of  information  service 
quality . 
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On  a  Likert  scale  ranging  from  2  to  6,  where  4  was  the 
median,  users  across  medical  centers  rated  their  level  of 
information  system  satisfaction  at  3.315.  The  range  of 
individual  medical  center  scores  was  from  3.158  to  3.418. 

There  were  no  discernible  trends  in  MSO  staffing 
sources  to  relate  to  the  level  of  user  satisfaction  at 
individual  medical  centers.  Most  MSOs  in  the  population  are 
composed  of  a  mixture  of  source  categories,  and  those  that 
are  made  up  of  predominantly  one  category  or  another  did  not 
produce  any  noticeable  pattern  in  satisfaction  rating.  In 
addition,  the  source  of  supervision  of  MSO  staff  members 
seemed  to  have  little  influence  on  the  satisfaction  ratings. 
The  MSO  staff  to  medical  center  staff  ratio  also  did  not 
appear  to  effect  the  level  of  information  service  quality. 

In  conclusion,  methods  of  staffing,  level  of  staffing, 
and  supervision  of  USAF  medical  center  systems  offices  seem 
to  be  independent  of  the  quality  of  the  services  provided  to 
their  customers  by  those  offices.  Therefore,  medical  sys¬ 
tems  office  staffing  should  be  tailored  to  meet  the  individ¬ 
ual  needs  of  the  medical  center  given  budget  and  other 
resource  constraints. 
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Appendix  A:  CIO  Responsibilities 

1.  Establish  local  implementation  of  higher  headquarters 
information  management  activities  involving  both  man¬ 
agement  and  clinical  information,  to  include  imple¬ 
mentation  plans,  site  preparation,  and  training. 

2.  Establish  local  information  management  activities  in 
conjunction  with  other  implemented  systems  to  ensure 
adequate  managerial  and  clinical  information  is  avail¬ 
able  for  all  levels  of  medical  treatment  facility  (MTF) 
management . 

3.  Employ  available  systems,  technology,  and  manpower  to 
provide  and  assist  with  the  analysis  and  presentation 
of  all  required  information. 

4.  Assist  the  MTF  Executive  Committee  to  identify  and 
determine  the  availability  of  needed  information 
resources . 

5.  Assist  other  formally  established  hospital  committees 
or  departments  in  the  acquisition,  analysis  and  presen¬ 
tation  of  required  information. 

6.  Appoint  a  Systems  Administrator  if  appropriate. 

7.  Prepare  the  annual  MTF  Communications-Computer  Systems 
Plan  for  the  facility. 

8.  Provide  input  for  the  Base  Communications-Computer 
Systems  Plan,  as  required. 

9.  Document  functional  information  system  requirements  in 
accordance  with  established  Major  Command  standards. 
Manuals  for  program  maintenance,  system  operations,  and 
users  must  be  prepared  for  all  applications  software 
developed  at  the  facility. 

10.  Coordinate  with  appropriate  base  agencies  such  as  the 
Communications  Squadron,  and  Civil  Engineering  on  all 
requirements  for  telecommunications  associated  with 
local  and  or  satellite  health  care  information  process¬ 
ing  systems. 

11.  Monitor  all  internally  developed  software  that  is 
undergoing  testing,  evaluation,  and  validation,  which 
must  be  documented  for  both  initial  operation  and  after 
subsequent  modifications  and  track  all  programmer 
actions  which  affect  any  production  programs  or  opera¬ 
ting  systems. 
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12.  Establish  procedures  to  assure  that  production  programs 
and  operating  systems  cannot  be  changed  without  know¬ 
ledge  and  approval  of  the  CIO  or  Database  Administra¬ 
tor  . 

13.  Provide  proper  management  controls  and  directives  for 
system  analysts  and  programmers  assigned  to  the  office, 
if  applicable  and  appoint  a  qualified  individual  to 
technically  review  all  software  programs  and  proce¬ 
dures  developed  on  the  local  systems.  Ensure  separa¬ 
tion  of  duties  where  the  operation  is  large  enough  to 
warrant  computer  operators,  programmers,'  analysts  and 
librarians.  Ensure  system  test  evaluations  are  per¬ 
formed  by  persons  other  than  the  system  developer. 

14.  Prepare  system  plans  for  security,  continuity  of  opera¬ 
tions,  routine  back-up  and  user  training.  Facilities 
must  .  .  .  appoint  a  Computer  Systems  Security  Officer 
and  Terminal  Area  Security  Officers  (TASO). 

15.  Certify  IS  equipment/services  received  for  payment. 

The  CIO  must  monitor  maintenance  performance. 

16.  Prepare  and  monitor  status  of  Systems  Change  Requests 
(SCR)  and  System  Incident  Reports  (SIR). 

17.  Coordinate  with  appropriate  Major  Command  Medical 
Informations  Systems  Offices,  [AFMSA,  and  other 
agencies]  on  all  matters  which  pertain  to  contractual 
arrangements  with  system  contractors. 

18.  Establish  and  chair  a  Medical  IS  Users’  Group.  Member¬ 
ship  should  include  representatives  from  each  func¬ 
tional  area  either  directly  using  or  deriving  benefits 
from  a  system.  The  purpose  of  this  group  is  to  define 
user  requirements  and  determine  responsibilities, 
problems  and  recommendati ons  relating  to  system  site 
preparation,  implementation,  operation  and  modifica¬ 
tion.  This  group  will  meet  at  the  discretion  of  the 
CIO,  but  at  least  quarterly.  Minutes  will  be  briefed 
to  the  MTF  Executive  Committee. 

19.  Establish  practices  and  procedures  to  protect  in-house 
IS  equipment  from  adverse  weather  conditions  such  as 
lightning  strikes  and  flood  damage  (AFR  168-4,  Chap  14, 
Para  14-5. f . (1))  . 
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Appendix  B:  Checklist  of  Control  Measures 


HEALTH  SERVICES  MANAGEMENT  INSPECTION  GUIDE  MARCH  1991 

YES  NO 


13.  Had  executive  management  committed  reasonable  fiscal 
and  manpower  resources  and  sufficient  visibility  within  the 
organization  to  provide  adequate  internal  understanding  and 
support  of  the  systems  installed  in  the  MTF?  (AFR  168-4, 

Para  15-5g(l)) 

SECURITY: 

14.  Were  Computer  Systems  Security  Officers  (CSSO)  and 
Terminal  Area  Security  Officers  (TASO)  designated  in  writ¬ 
ing?  (AFR  205-16,  Para  2-10,  2-11) 

15.  t/are  automated  data  processing  (ADP)  users  receiving 
annual  ADP  security  training?  (AFR  168-4,  Para  14-5f(l)(ni; 

AFR  205-16,  Para  16-8;  AFR  700-10,  Para  l-5m) 

16 .  Were  procedures  for  reporting  ADP  security  incidents 
■oi  fraud,  waste,  and  abuse  known  to  all  users?  (AFR  168-'i, 

Para  14-5f(l)(n);  AFR  205-16,  PAra  5-3f;  AFR  700-26,  Para  3-6b) 

17.  Did  each  individual  user  on  every  computer  system  have 
a  user  identification?  (AFR  168-4,  Para  14-5f(2)(a); 

AFR  205-15,  Para  8-2,  11-5) 

18.  Was  Privacy  Act  Information  protected  to  avoid  un¬ 
authorized  access  to  information  maintained  on  individual::? 

(AFR  700-10,  Para  2-5b) 

19.  Was  a  "Risk-Analysis *  of  all  comDUter  systems  (e.o., 
AQCESS ,  CHCS ,  MEDLOG,  MEPRS,  TMPS ,  TRILAB ,  etc.)  completed 
and  on  file?  (AFR  2G5-16,  Para  2-Se,  Atch  8,  Para  A8-7) 

20.  Was  there  a  current  security  plan?  (AFR  168-4, 

Paral4-5f ( 1) (n) ;  AFR  205-16,  Atch  7) 

21.  If  processing  classified  information: 

a.  Was  a  TEMPEST  check  of  all  equipment  used  to  pro¬ 
cess,  store  or  display  classified  information  performed 
annually  with  the  last  two  tests  kept  on  file? 

(AFR  125-37,  Para  15-le;  AFR  700-10,  Para  2-8) 

b.  Was  access  to  the  computer  restricted  only  to 
personnel  who  have  a  need  to  use  it?  (AFR  205-16,  Para 
8-2;  AFR  700-10,  Para  2-8a) 

22.  Had  the  security  police  and/or  Resource  Protection 
Executive  Committee  made  a  determination  as  to  whether  or 
not  the  computer  room  needed  to  be  a  controlled  area? 

(AFR  125-37,  Para  12-2c(3),  15-lc;  AFR  700-7,  Para  2-8a) 

23.  Was  entry  to  the  computer  room  controlled? 

(AFR  125-37,  Para  15-lb) 
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HEALTH  SERVICES  MANAGEMENT  INSPECTION  GUIDE 


MARCH  1991 


YES  NO 


13.  Had  executive  management  committed  reasonable  fiscal 
and  manpower  resources  and  sufficient  visibility  within  the 
organization  to  provide  adequate  internal  understanding  and 
support  of  the  systems  installed  in  the  MTF?  (AFR  166-4, 

Para  15-5g ( 1 ) ) 

SECURITY: 

14.  Were  Computer  Systems  Security  Officers  (CSSO)  and 
Terminal  Area  Security  Officers  (TASO)  designated  in  writ¬ 
ing?  (AFR  205-16,  Para  2-10,  2-11) 

15.  f-.'tre  automated  data  processing  (ADP)  users  receiving 
annual  ADP  security  training?  (AFR  168-4,  Para  14-5f ( 1) (nl ; 

AFR  205-16,  Para  16-8;  AFR  700-10,  Para  l-5m) 

16.  Were  procedures  for  reporting  ADP  security  incidents 
of  fraud,  waste,  and  abuse  known  to  all  users?  (AFR  168-'., 

Para  14-5f(l)(n);  AFR  205-16,  Para  5-3f;  AFR  700-26,  Para  3-6b) 

17.  Did  each  individual  user  on  every  computer  system  have 
a  user  identification?  (AFR  168-4,  Para  14-5f(2)(a); 

AFR  2C5-16 ,  Para  8-2,  11-5) 

13.  Was  Privacy  Act  Information  protected  to  avoid  un¬ 
authorized  access  to  information  maintained  on  individual:;? 

(AFR  700-10,  Para  2-5b) 

19.  Was  a  "Risk-Analysis’  of  all  comouter  systems  (e.o., 
AQCESS,  CHCS ,  MEDLOG,  MEPRS,  TMPS,  TRILAB,  etc.)  completed 
and  on  file?  (AFR  2C5-16,  Para  2-Se,  Atch  8,  Para  A8-7) 

20.  Was  there  a  current  security  plan?  (AFR  168-4, 

Paral4-5f ( 1) (n) ;  AFR  205-16,  Atch  7) 

21.  If  processing  classified  information: 

a.  Was  a  TEMPEST  check  of  all  equipment  used  to  pro¬ 
cess,  store  or  display  classified  information  performed 
annually  with  the  last  two  tests  kept  on  file? 

(AFR  125-37,  Para  15-le;  AFR  700-10,  Para  2-8) 

b.  Was  access  to  the  computer  restricted  only  to 
personnel  who  have  a  need  to  use  it?  (AFR  205-16,  Para 
8-2;  AFR  700-10,  Para  2-8a) 

22.  Had  the  security  police  and/or  Resource  Protection 
Executive  Committee  made  a  determination  as  to  whether  or 
not  the  computer  room  needed  to  be  a  controlled  area? 

(AFR  125-37,  Para  12-2c(3),  15-lc;  AFR  700-7,  Para  2-8a) 

23.  Was  entry  to  the  computer  room  controlled? 

(AFR  125-37,  Para  15-lb) 
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HEALTH  SERVICES  MANAGEM  T  INSPECTION  GUIDE 


MARCH  1991 


CONTINGENCY  PLANNING: 


24 .  Had  contingency  plans  been  developed  to  support  opera¬ 
tions  under  emergency  conditions?  (APR  700-6,  Para  l-4d; 
APR  700-7,  Para  2-11  through  2-14;  APR  700-26,  Para  2-11, 

3- 6a(2) ) 

25.  Were  the  contingency  plans  exercised  at  least  annu¬ 
ally?  (APR  700-7,  Para  l-5m) 

STORAGE  MEDIA  MANAGEMENT: 

26.  Was  magnetic  media  within  the  facility  controlled 
adequately  to  avoid  loss,  theft,  etc.?  (AFR  700-7,  Para 

4- 2;  AFR  700-26,  Para  2-11) 

27.  Were  ADP  storage  media  marked  according  to  sensitivity 
or  classification?  (APR  205-16,  Para  2-12b,  13-4d) 

28.  Was  the  fire  department  aware  of  magnetic  media  stor¬ 
age  locations?  (AFOSH  Std  127-64) 

29.  Had  a  suitable  off-site  backup  storage  location  and 
rotation  cycle  been  established  for  magnetic  media  and 
critical  supplies?  (AFR  700-7,  Para  2-i6b;  AFR  700-26, 

Para  2-11) 

FACILITY  MANAGEMENT  (ENVIRONMENT): 

30.  Were  work  areas  clean,  free  of  boxes  and  other  clut¬ 
ter?  (AFR  700-7,  Para  2-8e) 

31.  Was  smoking,  eating,  and  drinking  disallowed  in  the 
main  computer  room?  (AFR  700-7,  Para  2-8e(5)) 

32.  Were  emergency  shut-offs  for  each  main  computer  room 
located  at  each  fire  exit?  (AFOSH  Std  127-64) 

33.  Were  circuit  breakers  in  the  power  panels  marked  with 
individual  pieces  of  equipment  tied  into. them?  (AFOSH  Std 
127-64) 

34 .  Was  the  computer  room  environment  adequate  in  terms  of 
temperature  humidity  and  dust  control?  (AFR  700-7,  Para 
2-8b) 

35.  Were  thermometers /hygrometers  calibrated  semiannually 
and  were  the  charts  properly  annotated  and  maintained  for  3 
years?  (AFM  88-15) 

36 .  Were  hand-held  fire  extinguishers  located  in  the  com¬ 
puter  room  and  were  they  easily  accessible  to  the  opera¬ 
tors?  (AFOSH  Std  127-64) 
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HEALTH  SERVICES  MANAGEMENT  INSPECTION  GUIDE 


MARCH  1991 


YES  NO 


3?.  w«re  abort  switches  installed  for  control'ed  abort  of 
automatic  discharge  of  a  total  flooding  Halon  fire  suppres¬ 
sion  system  if  installed?  (AFM  88-15,  Para  15-75(4)  (c)) 
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Appendix  C:  CIO  Survey 


MEDICAL  CENTER _ 

INFORMATION  SYSTEM  OFFICER  NAME 

PHONE  NUMBER _ 

NUMBER  ON  MEDICAL  CENTER  STAFF_ 

QUESTIONNAIRE  DISTRIBUTION  POC_ 

PHONE  #_ 


IS  OFFICE  DEMOGRAPHICS 
NUMBER  ON  IS  OFFICE  STAFF _ 


SOURCE  OF  STAFF: 

PROGRAMMERS 

#  906XXW  _ 

#  49XXX  _ 

#  MED  CTR  SUPERVISED  _  #  SC  SUPERVISED  _ 

#  CIVILIANS  _ 

#  CONTRACTED  _ 

#  CIVIL  SERVICE  _ 

OCCUPATIONAL  CODE  TITLE  AND  NUMBER  OF  EACH 
AVE  NUMBER  OF  YEARS  OF  EXPERIENCE _ 

LAN  SPECIALISTS 

#  906XXW  _ 

#  49XXX  _ 

#  MED  CTR  SUPERVISED  _  #  SC  SUPERVISED  _ 

#  CIVILIANS  _ 

#  CONTRACTED  _ 

#  CIVIL  SERVICE  _ 

OCCUPATIONAL  CODE  TITLE  AND  NUMBER  OF  EACH 
AVE  NUMBER  OF  YEARS  OF  EXPERIENCE  _ 

CUSTOMER  SERVICE 

#  906XXW  _ 

#  49XXX  _ 

#  MED  CTR  SUPERVISED  _  #  SC  SUPERVISED  _ 

#  CIVILIANS  _ 

#  CONTRACTED  _ 

#  CIVIL  SERVICE  _ 

OCCUPATIONAL  CODE  TITLE  AND  NUMBER  OF  EACH 
AVE  NUMBER  OF  YEARS  OF  EXPERIENCE  _ 


INDIVIDUAL  SYSTEM  SPECIALISTS 

SYSTEM _ 

#  906XXW  _ 

#  49XXX  _ 

#  MED  CTR  SUPERVISED _ #  SC  SUPERVISED  _ 

#  CIVILIANS  _ 

#  CONTRACTED  _ 

#  CIVIL  SERVICE  _ 

OCCUPATIONAL  CODE  TITLE  AND  NUMBER  OF  EACH 
AVE  NUMBER  OF  YEARS  OF  EXPERIENCE  _ 

SYSTEM _ 

#  906XXW  _ 

#  49XXX  _ 

#  MED  CTR  SUPERVISED  _  #  SC  SUPERVISED  _ 

#  CIVILIANS _ 

#  CONTRACTED  _ 

#  CIVIL  SERVICE  _ 

OCCUPATIONAL  CODE  TITLE  AND  NUMBER  OF  EACH 
AVE  NUMBER  OF  YEARS  OF  EXPERIENCE  _ 

SYSTEM _ 

#  906XXW  _ 

#  49XXX  _ 

#  MED  CTR  SUPERVISED  _  #  SC  SUPERVISED  _ 

#  CIVILIANS  _ 

#  CONTRACTED  _ 

#  CIVIL  SERVICE  _ 

OCCUPATIONAL  CODE  TITLE  AND  NUMBER  OF  EACH 
AVE  NUMBER  OF  YEARS  OF  EXPERIENCE  _ 

SYSTEM _ 

#  906XXW  _ 

#  49XXX  _ 

#  MED  CTR  SUPERVISED  _  #  SC  SUPERVISED  _ 

#  CIVILIANS  _ 

#  CONTRACTED  _ 

#  CIVIL  SERVICE  _ 

OCCUPATIONAL  CODE  TITLE  AND  NUMBER  OF  EACH 
AVE  NUMBER  OF  YEARS  OF  EXPERIENCE  _ 

SYSTEM _ 

#  906XXW  _ 

#  49XXX  _ 

#  MED  CTR  SUPERVISED  _  #  SC  SUPERVISED  _ 

#  CIVILIANS  _ 

#  CONTRACTED  _ 

#  CIVIL  SERVICE  _ 

OCCUPATIONAL  CODE  TITLE  AND  NUMBER  OF  EACH 
AVE  NUMBER  OF  YEARS  OF  EXPERIENCE  _  __ 
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#  SC  SUPERVISED 


SYSTEM _ 

#  906XXW  _ 

#  49XXX  _ 

#  MED  CTR  SUPERVISED 

#  CIVILIANS  _ 

#  CONTRACTED  _ 

#  CIVIL  SERVICE  _ 

OCCUPATIONAL  CODE  TITLE  AND  NUMBER  OF  EACH 
AVE  NUMBER  OF  YEARS  OF  EXPERIENCE  _ 

MANAGEMENT 

#  MILITARY  MSC  _  RANK  _ 

#  CIVILIANS  _ 

#  CONTRACTED  _ 

#  CIVIL  SERVICE  _  GRADE  _ 

OCCUPATIONAL  CODE  TITLE  AND  NUMBER  OF  EACH 
AVE  NUMBER  OF  YEARS  OF  EXPERIENCE  _ 

SYSTEMS  INFORMATION 


NAME  OF  SYSTEM _ 

NUMBER  OF  YEARS  ON  LINE 
APPLICATION  _ 

NAME  OF  SYSTEM  _ 

NUMBER  OF  YEARS  ON  LINE 
APPLICATION  _ 

NAME  OF  SYSTEM  _ 

NUMBER  OF  YEARS  ON  LINE 
APPLICATION  _ 

NAME  OF  SYSTEM  _ 

NUMBER  OF  YEARS  OI  LINE 
APPLICATION  _ 

NAME  OF  SYSTEM  _ 

NUMBER  OF  YEARS  ON  LINE 
APPLICATION  _ 

NAME  OF  SYSTEM  _ 

NUMBER  OF  YEARS  ON  LINE 
APPLICATION  _ 


CONTROL  STATISTICS 

MEASUREMENT  _ 

FREQUENCY  OF  MEASUREMENT  _ 

LAST  SIX  MONTHS  DATA 


MEASUREMENT  _  . 

FREQUENCY  OF  MEASUREMENT 
LAST  SIX  MONTHS  DATA 


MEASUREMENT  _ 

FREQUENCY  OF  MEASUREMENT 
LAST  SIX  MONTHS  DATA 


MEASUREMENT _ 

FREQUENCY  OF  MEASUREMENT 
LAST  SIX  MONTHS  DATA 


MEASUREMENT  _ 

FREQUENCY  OF  MEASUREMENT 
LAST  SIX  MONTHS  DATA 


MEASUREMENT  _ 

FREQUENCY  OF  MEASUREMENT 
LAST  SIX  MONTHS  DATA 
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INFORMATION  SYSTEM  BUDGET 


FISCAL  YEAR  1992  BUDGET  _ 

NEW  SYSTEMS  CURRENTLY  BEING  IMPLEMENTED 

SYSTEM _ 

BUDGETED  AMOUNT  _ _ 

SYSTEM _ 

BUDGETED  AMOUNT  _ 

BREAKDOWN  OF  BUDGET  APPROPRIATION  ITEMS 

ITEM  TITLE _ 

BUDGETED  AMOUNT  _ 

ITEM  TITLE _ 

BUDGETED  AMOUNT _ 

ITEM  TITLE _ 

BUDGETED  AMOUNT  _ 

ITEM  TITLE _ 

BUDGETED  AMOUNT  _____ _ 

ITEM  TITLE _ 

BUDGETED  AMOUNT  _ 

ITEM  TITLE _ 

BUDGETED  AMOUNT  _ 


Appendix  D:  Information  System  Users  Survey 


SURVEY  INSTRUCTIONS 

Please  completely  fill  in  your  answers  to  questions  01  through  61  on  your  computer  answer 
sheet  WITH  A  NUMBER  2  PENCIL 

EXAMPLE: 


For  those  questions  requiring  a  written  answer,  please  respond  on  the  survey  itself. 

Please  do  not  write  your  name  on  either  the  answer  sheet  or  the  survey.  Please  avoid  any 
stray  marks  on  the  computer  answer  sheet  and  do  not  fold  the  computer  answer  sheet 

After  you  have  completed  the  survey,  please  place  your  SURVEY  and  your  ANSWER 
SHEET  in  the  envelope  provided  and  return  them  to  us  through  your  medical  center’s 
distribution  system.  No  postage  s  necessary. 

THANK  YOU  TOR  YOUR  PARTICIPATION!! 
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INFORMATION  SYSTEMS  QUESTIONNAIRE 


PLEASE  COMPLETELY  FILL  IN  YOUR  ANSWERS  TO  QUESTIONS  01  THROUGH  61  ON  YOUR 
COMPUTER  ANSWER  SHEET  WITH  NUMBER  2  PENCIL. 

PART  1 

01.  Indicate  your  sex. 

1.  Female. 

2.  Male. 

02.  Find  the  choice  nurber  below  that  includes  your  age. 


1. 

25 

or 

less. 

2. 

26 

to 

32. 

3. 

33 

to 

39. 

4. 

40 

to 

46. 

5. 

47 

to  53. 

6. 

54 

to 

60. 

7. 

61 

or 

more. 

03.  Indicate  the  highest  level  of  education  you  have  achieved. 

1.  Did  not  complete  high  school. 

2.  High  school  graduate  or  equivalent. 

3.  Seme  courses  beyond  high  school  but  no  degree  awarded. 

4.  Associate's  Degree,  technical  or  trade  school  certificate. 

5.  Bachelor's  Degree. 

6.  Sane  graduate  courses. 

7.  Master's  Degree  or  higher. 

04.  Select  the  nurber  below  that  describes  your  status. 

1.  Active  duty  military  (go  to  question  #  05). 

2.  Civilian  civil  service  (go  to  question  #  06). 

3.  Civilian  other  (go  to  question  #07). 

05.  (Active  duty  military  only)  Find  the  nurber  below  that  includes  your 
rank. 

1.  E-l  to  E-4  (go  to  question  #  10). 

2.  E-5  to  E-6  (go  to  question  #  10). 

3.  E-7  to  E-9  (go  to  question  #  10). 

4.  0-1  to  0-3  (go  to  question  t  08). 

5.  0-4  to  0-5  (go  to  question  #08). 

6.  0-6  or  higher  (go  to  question  #08). 

06.  (Civilian  civil  service  only)  Find  the  nurber  below  that  includes  your 
grade. 

1.  Wage  grade  (go  to  question  #  10). 

2.  GS-1  to  GS-4  (go  to  question  #  10). 

3.  GS-5  to  GS-8  (go  to  question  #  10). 

4.  GS-9  to  GS-12  (go  to  question  #  10). 

5.  GS/GM-13  or  higher  (go  to  question  #  10). 
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07.  (Civilian  noncivil  service  only)  How  would  you  describe  your  status  with 
the  medical  center? 

1.  contracted  service  (go  to  question  #  10). 

2.  volunteer  (go  to  question  #10). 

3.  other.  Please  specify _ . 

(go  to  question  #10). 

08.  (Active  duty  officers  only)  To  which  corps  do  you  belong? 

1.  Medical  Corps  (go  to  question  #  10). 

2.  Dental  Corps  (go  to  question  #10). 

3.  Nurse  Corps  (go  to  question  #  09). 

4.  Medical  Service  Corps  (go  to  question  #  10). 

5.  Biomedical  Service  Corps  (go  to  question  #  10). 

6.  None  of  the  above.  I  am  a  nonmedical  officer  (go  to  question  #  10). 

09.  (Nurse  Corps  only)  Where  do  you  spend  the  majority  of  your  time  perform¬ 
ing  your  duties? 

1.  In  an  in-patient  setting. 

2.  In  an  out-patient  setting. 

3.  My  time  is  spent  approximately  equally  in  both  settings. 

10.  Select  the  statement  below  that  best  describes  your  work. 

1.  Direct  inpatient  care  duties. 

2.  Direct  outpatient  care  duties  in  a  clinic. 

3.  Direct  patient  care  duties  in  an  ancillary  service 
(pharmacy,  laboratory,  physical  therapy,  etc.). 

4.  Medical  administration  duties. 

5.  Duties  related  to  medical  logistics. 

6.  Duties  not  specified  above. 

Please  specify.  _ 

11.  How  long  have  you  used  a  computer  on  or  off  the  job? 

1.  Less  than  1  year. 

2.  1  to  3  years. 

3.  3  to  5  years. 

4.  5  to  10  years. 

5.  10  years  or  more. 

12.  Indicate  the  highest  level  of  computer  training  you  have  completed. 

1.  None. 

2.  Self  taught. 

3.  High  school  course(s). 

4.  Adult,  continuing  education. 

5.  College  course(s) 

6.  Associate  Degree  in  Computer  Science,  Information  Resource 
Management,  or  Management  Information  Systems. 

7.  Bachelor  Degree  in  Computer  Science,  Information  Resource 
Management,  or  Management  Information  Systems. 

8.  Master  Degree  or  higher  in  Computer  Science,  Information  Resource 
Management,  or  Management  Information  Systems. 

9.  Other.  Please  specify _ . 


13.  How  long  have  you  been  in  your  current  assignment  at  the  medical  center? 

1.  0  to  1  year. 

2.  1  to  2  years. 

3.  2  to  3  years. 

4.  3  to  4  years. 

5.  more  than  4  years. 

14.  How  long  have  you  been  in  your  current  occupation  or  profession? 

1.  0  to  5  years. 

2.  5  to  10  years. 

3.  10  to  IS  years. 

4.  15  to  20  years. 

5.  more  than  20  years. 

For  Questions  15  through  23,  rate  how  confident  you  feel  using  that 
application  in  your  duties  at  the  medical  center.  Use  the  following  scale  for 
each  question. 

1.  1  do  not  recognize  this  application. 

2.  I  do  not  use  it  at  all. 

3.  I  can  perform  only  basic  functions  following  prorpts  or  menus; 

I  usually  need  help  recovering  from  mistakes . 

4.  I  can  perform  all  of  the  basic  functions  and  follow  instructions 
in  a  manual  for  more  advanced  functions;  I  sometimes  require  help 
in  performing  the  more  advanced  functions. 

5.  1  can  perform  all  of  the  basic  and  advanced  functions;  I  rarely, 
if  ever,  require  assistance. 

6.  1  can  perform  all  of  the  functions  of  the  application;  others  seek 
my  help  in  using  the  application. 

15.  Word  Processing. 

16.  Graphics. 

17.  Electronic  Mail  (email). 

18 .  Appointment  System. 

19.  Applications  specific  to  my  department. 

20.  Medical  Records  Access. 

21.  Report  Production. 

22.  Decision  Support  Systems. 

23.  Spreadsheets. 
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For  Questions  24  through  28,  specify  and  rate  your  confidence  with  other 
applications  you  use  in  your  duties  at  the  medical  center  using  the  choices 
listed  for  Questions  15  through  23.  If  you  dp  not  use  other  applications  in 
your  duties,  leave  Questions  24  through  28  blank  an  your  answer  sheet. 

1.  I  do  not  recognize  this  application. 

2.  I  do  not  use  it  at  all. 

3.  I  can  perform  only  basic  functions  following  prcrrpts  or  menus; 

I  visually  need  help  recovering  from  mistakes. 

4.  1  can  perform  all  of  the  basic  functions  and  follow  instructions 
in  a  manual  for  more  advanced  functions;  I  sometimes  require  help 
in  performing  the  more  advanced  functions. 

5.  I  can  perform  all  of  the  basic  and  advanced  functions;  I  rarely, 
if  ever,  require  assistance. 

6.  I  can  perform  all  of  the  functions  of  the  application;  others  seek 
my  help  in  using  the  application. 

24.  Other.  Please  specify _ . 

25.  Other.  Please  specify _ . 

26.  Other.  Please  specify _ . 

27.  Other.  Please  specify _ . 

28.  Other.  Please  specify  -  . 

29.  Select  the  statement  which  most  accurately  describes  your  use  of  the 
available  applications  at  the  medical  center. 

1.  Most  often,  I  directly  access  a  computer  terminal,  keying  in 
information  myself. 

2.  Most  often,  I  directly  access  a  conputer  terminal,  keying  in 
information  and  producing  reports  myself. 

3.  Most  often,  I  do  not  directly  access  a  computer  terminal,  but  rather 
use  the  information  keyed  in  by  someone  else. 

4.  Most  often,  1  do  not  directly  access  a  conputer  terminal,  but  rather 
use  the  information  and  reports  produced  by  someone  else. 

5.  I  directly  access  a  conputer  terminal,  keying  in  information  and 
producing  reports  about  as  often  as  I  use  the  information  and 
reports  keyed  in  by  someone  else. 

6.  Mane  of  the  above.  1  do  not  access  the  conputer  terminal,  nor  do  I 
use  the  information  and  reports  keyed  in  by  someone  else. 

30.  Has  using  a  conputer  improved  the  QUALITY  of  your  work? 

1.  I  don’t  know. 

2.  Very  little  improvement. 

3.  Little  improvement . 

4.  Moderate  improvement . 

5.  Much  improvement. 

6.  Very  much  improvement. 
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31.  Has  using  a  ccnputer  inproved  the  QUANTITY  of  work  you  are  able  to  do? 

1.  I  don’t  know. 

2.  Very  little  increase. 

3.  Little  increase. 

4.  Moderate  increase. 

5.  Much  increase. 

6.  Very  much  increase. 

32.  How  much  confidence  do  you  have  using  medical  center  computers  to 
meet  the  needs  of  your  job? 

1.  I  don't  know. 

2.  Very  low  confidence. 

3.  Low  confidence. 

4.  Moderate  confidence. 

5.  High  confidence. 

6.  Very  high  confidence. 

33.  Do  you  feel  that  you  could  be  more  productive  in  your  job  if  you  had 
stronger  computer  skills? 

1.  I  don’t  know. 

2.  Very  little  more. 

3.  A  little  more. 

4.  More. 

5.  Much  more. 

6.  Very  much  more. 
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PART  2 

Fran  the  following  list  please  choose  the  three  information  systems  you 
use  most  in  performing  your  duties  at  the  medical  center.  Please  put  your 
answers  on  the  questionnaire.  If  you  use  fewer  than  three  systems,  leave  the 
remaining  ones  blank. 

Example:  If  Tripharm  is  one  of  the  systems  you  use  most, 
write  a  "4"  in  the  blank  to  the  right  of  "SYSTEM  #1" 
below.  The  blanks  to  the  right  of  "SYSTEM  #2"  and 
"SYSTEM  #3"  should  be  filled  in  with  the  numbers  of 
two  other  systems  you  use  most  in  performing  your 
duties. 

SYSTEMS  LIST 

1.  Aqcess 

2.  Managed  Health  Care  (ATT) 

3.  Trilab 

4.  Tripharm 

5.  Trirad 

6.  Wang  Office 

7.  Wang  PAS  (Appointment  System) 

8.  CAPOC 
«.  MEPERS 

10.  EAS  III 

11.  **MS 

12.  PCIII 

13.  MEDLOG 

14.  MEDNET 

15.  CHCS  (Malcolm  Grow  only) 

16.  CMIS  (Xeesier  only) 

17.  LAB  (Wilford  Hall  only) 

18.  Local  area  network 

19.  Sun  Workstations 

20.  Other  (please  write  the  system  name) _ _ 

21.  Other  (please  write  the  system  name) _ 

22.  Other  (please  write  the  system  name) _ 


SYSTEM  #1  _ 

(This  will  be  referred  to  as  SYS1  in  Questions  34  through  57) 
SYSTEM  #2  _ 

(This  will  be  referred  to  as  SYS2  in  Questions  34  through  57) 
SYSTEM  13  _ 

(This  will  be  referred  to  as  SYS3  in  Questions  34  through  57) 


1  C  1 
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This  part  of  the  survey  examines  the  types  of  computer  support  you  may  require 
in  performing  your  duties  at  the  medical  center  and  the  importance  of  that 
type  of  support.  PLEASE  USE  THE  THREE  SYSTEMS  YOU  SELECTS}  ON  THE  PREVIOUS 
PAGE  TO  ANSWER  QUESTIONS  34  THROUGH  57  (REFERRED  TO  AS  SYS1,  SYS2,  AND  SYS3). 
There  are  4  support  areas  about  which  you  will  be  asked.  Each  support  area  is 
identified  in  bold  letters  followed  by  its  definition. 

To  answer  the  questions  in  this  section  of  the  questionnaire,  please  use  the 
following  scale  to  nark  your  answers  on  the  answer  sheet. 

1 - 2 - 3 - 4 - 5 - 6 

no  very  low  moderate  high  very 

opinion  low  high 

DEFINITION:  PERFORMANCE  is  a  measure  of  how  well  you  think  the  computer 
system  does  what  you  think  it  should  do,  such  as  readability  and  usefulness  of 
reports  and  printouts,  availability  of  necessary  information,  and  correctness 
of  output. 

Please  rate  your  level  of  satisfaction  with  the  performance  of  each  of  the 
three  systems  you  selected. 

34.  SYS1 

35.  SYS2 

36.  SYS3 

How  much  importance  do  you  place  an  the  performance  of  each  of  the  three 
systems  you  selected? 

37.  SYS1 

38 .  SYS2 

39.  SYS3 

DEFINITION:  ADDITIONAL  FEATURES  sure  those  options  that  add  to  the  ccnputer 
system's  functionality,  but  are  not  required  in  performing  your  duties  at  the 
medical  center.  Examples  of  additional  features  include  menus,  on-line  help, 
and  virus  scanning. 

Please  rate  your  level  of  satisfaction  with  the  additional  features  of  each  of 
the  three  systems  you  selected. 

40.  SYS1 

41.  SYS2 

42.  SYS3 

How  much  importance  do  you  place  an  the  additional  features  of  each  of  the 
three  systems  you  selected? 

43.  SYS1 

44.  SYS2 

45.  SYS3 
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1 - 2 - 3 - 4 - 5 - 6 

no  very  low  moderate  high  very 

opinion  low  high 

DEFINITION:  SERVICEABILITY  is  the  amount  of  time  the  computer  system  is  in 
full  service  to  the  users  and  the  degree  to  which  the  medical  information 
system  specialists  provide  prompt,  knowledgeable,  and  courteous  maintenance. 

Please  rate  your  level  of  satisfaction  with  the  serviceability  of  each  of  the 
three  systems  you  selected. 

46.  SYS1 

47 .  SYS2 

48.  SYS3 

How  much  importance  do  you  place  on  the  serviceability  of  each  of  the  three 
systems  you  selected? 

49.  SYS1 
5d .  SYS2 

51.  SYS3 

DEFINITION:  AESTHETICS  are  a  measure  of  the  degree  the  computer  system 
appeals  to  your  senses  in  such  things  as  monitor  screen  colors,  appearance  of 
reports  and  printouts,  location  of  computer  terminals,  and  operator  comfort  at 
computer  work  stations. 

Please  rate  your  level  of  satisfaction  with  the  aesthetics  of  each  of  the 
three  systems  you  selected. 

52 .  SYS1 

53 .  SYS2 

54.  SYS3 

How  much  importance  do  you  place  an  the  aesthetics  of  each  of  the  three 
systems  you  selected? 

55 .  SYS1 

56.  SYS2 

57 .  SYS3 
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PART  3 

58.  Please  rate  your  level  of  confidence  in  the  medical  center’s  information 
system  office  staff. 

1.  Mo  opinion. 

2.  Very  Low. 

3.  Low. 

4.  Moderate. 

5.  High. 

6.  Very  High. 


59.  Please  rate  your  level  of  satisfaction  with  any  assistance  you  may 

have  received  from  the  medical  center's  information  system  office  staff 
with  computer- related  problems  or  questions. 

1.  I  have  not  received  any  assistance. 

2.  Very  Low. 

3.  Low. 

4.  Moderate. 

5.  High. 

6.  Very  High. 

6(5.  Please  rate  your  level  of  satisfaction  in  the  medical  center  computer 
system's  ability  to  help  you  perform  your  job. 

1.  No  opinion. 

2.  Very  Low. 

3.  Low. 

4.  Moderate. 

5.  High. 

6.  Very  High. 

61.  At  which  USAF  Medical  Center  do  you  perform  your  duties? 

1.  David  Grant  Medical  Center 

2.  Keesler  AFB  Medical  Center 

3.  Malcolm  Grow  Medical  Center 

4.  Scott  AFB  Medical  Center 

5.  Wilford  Hall  Medical  Center 

6.  Wright-Pattersan  AFB  Medical  Center 

62.  Please  provide  below  any  additional  comments  pertaining  to  the  medical 
center  computer  system's  quality  and  the  quality  of  service  provided  to 
you  by  the  medical  center  information  system  staff  that  you  think  might 
be  useful. 


THANK  YOU  FOR  YOUR  VALUABLE  PARTICIPATION! 


Appendix  E:  Survey  Distribution  Instructions 


INSTRUCTIONS  FOR  SURVEY  DISTRIBUTION 

FOR  MEDICAL  SYSTEMS  OFFICER 

Give  each  of  your  TASOs  1  survey  per  7  users  assigned  to  them. 
Please  round  this  number  u£  to  the  nearest  whole  number. 

Please  ask  your  TASOs  to  distribute  the  surveys  as  soon  as  they 
possibly  can  following  the  instructions  below.  We  would 
appreciate  your  encouragement  of  the  TASOs  to  solicit  maximum 
participation. 

FOR  TASOs 

Please  choose  any  number  between  i  and  7.  Begin  selection  of 
survey  participants  with  the  name  of  that  person  on  your  list  of 
users  assigned  to  you  corresponding  to  the  number  you  have 
chosen.  Then  give  one  survey  to  every  7th  name  on  the  list 
counting  from  the  first  name. 

For  example,  if  you  chose  the  number  U,  distribute  surveys  to  the 
fcth,  Ilth,  I8th,  25th,  32nd  people  on  your  list  until  the  list  is 
exhausted. 

If  you  have  exhausted  your  list  and  you  still  have  surveys  left, 
choose  another  number  between  1  and  7  and  begin  again  until  all 
surveys  are  distributed. 

Once  the  surveys  have  been  completed,  the  participants  should 
place  their  computer  answer  sheets  and  surveys  in  the  envelopes 
provided  and  send  them  to  us  through  distribution  channels. 

Please  encourage  maximum  participation.  Also,  we  would 
appreciate  responses  as  soon  as  possible. 

THANK  YOU  FOR  YOUR  HELP  IN  OUR  RESEARCH 


Appendix  F:  Information  System  Summaries 


Managed  Health  Care 

Managed  Health  Care  was  contracted  with  Computer  Data 
Systems,  Inc.,  for  development  and  implementation  at  Wright- 
Patterson  USAF  Medical  Center.  It  first  became  operational 
in  1991.  Its  functions  are  in  three  categories: 

1.  Program  Management  -  health  care  finders.  Veteran's 
Administration  sharing,  partnership  agreements, 
marketing  and  recruitment 

2.  Beneficiary  Referrals  -  CHAMPUS,  tracking  referrals 
and  disengagements,  management  indicators,  catch¬ 
ment  area  tracking,  area-wide  medical  service 
tracking 

3.  Alternative  Care  -  appointments  tracking,  budgeting, 
in-house  clinic  service  availability  and  waiting 
periods . 

The  system  does  not  currently  interface  with  any  other 
systems  and  no  proliferation  among  other  Air  Force  facili¬ 
ties  is  planned.  Some  of  Managed  Health  Care’s  function¬ 
ality  will  be  taken  over  by  CHCS  (Wheel er , 1992 ) . 


Commander's  Medical  Information  System  (CMIS) 

This  information  system  was  developed  by  in-house  staff 
members  for  implementation  at  Keesler  USAF  Medical  Center. 

It  was  designed  to  automate  many  personnel  functions  previ¬ 
ously  performed  manually.  Its  functions  include:  personnel 
management  and  orderly  room  functions,  medical  readiness 
(training  tracking,  mobility  and  recall  management),  nursing 
management  (credentials,  training  and  continuing  medical 
education  tracking,  nurse  staffing),  public  health  tracking, 
and  hospital  services  (supplemental  care  and  payments). 
Additionally,  it  provides  cytology-histology  reporting  and 
genetics  clinic  reporting  and  tracking. 

It  is  capable  of  on-line  real-time  reporting  of  labora¬ 
tory  test  results  to  remote  sites  via  modem.  Future  plans 
for  this  system  include  CAP  tracking  for  pathology,  inter¬ 
face  with  PCIII,  and  conversion  to  open  architectural  envi¬ 
ronment  for  implementation  at  all  Air  Training  Command  bases 
(Williams , 1992 ) . 


Regen  Strief  Clinical  Laboratory  System  (LAB) 

This  system  was  adapted  for  use  at  Wilford  Hall  USAF 
Medical  Center  by  The  Regen  Strief  Institute  from  a  similar 
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system  at  Wishard  Memorial  Hospital  in  Indianapolis.  It 
became  operational  at  Wilford  Hall  in  1982.  Two  of  its 
modules,  the  gynecology  cytology  module  and  anatomical 
physiology  module,  were  programmed  and  developed  by  Wilford 
Hall  pathology  department  staff  members.  Through  the  use  of 
barcoding,  it  can  unidirectional ly  and  bidirectionally 
interface  with  laboratory  test  instruments.  It  is  capable 
of  database  searches  for  research  purposes,  electronic 
result  transfer  via  modem  interface  to  60  remote  locations, 
and  some  laboratory  CAP  workload  tracking.  It  does  not 
handle  ward  order  entry  or  donor  center  and  blood  bank 
functions . 

Future  plans  include  automatic  coupling  without  human 
intervention.  There  are  no  plans  for  the  proliferation  of 
this  system  to  other  Air  Force  facilities  (Mil  Is , 1992 ) . 


Composite  Health  Care  System  (CHCS) 

The  CHCS  is  a  comprehensive  and  fully  integrated 
information  support  system  designed  to  enhance  the  quality 
of  care  provided  in  military  medical  treatment  facilities  by 
providing  automated  clinical  support  to  health  care  provid¬ 
ers  and  administrators.  It  supports  functional  work  centers 
throughout  the  hospital  including  pharmacy,  laboratory, 
patient  administration,  patient  appointment  and  scheduling, 
nursing,  clinical  dietetics,  anatomical  pathology,  and  blood 
bank.  This  system  will  interface  with  other  standard  DOD 
and  service-specific  medical  automation  initiatives  includ¬ 
ing  medical  logistics  (MEDLOG),  food  service  (Trifood), 
DEERS,  and  the  Medical  Expense  and  Performance  Reporting 
System  (MEPRS) . 

The  first  Air  Force  beta  testing  sites  of  CHCS  were  at 
Sheppard  AFB,  Keelser  AFB ,  and  Eglin  AFB .  Presently,  some 
of  the  CHCS  modules  are  still  undergoing  beta  testing  while 
others  have  been  approved  for  full  implementation.  Imple¬ 
mentation  is  planned  at  all  Air  Force  medical  treatment 
facilities,  with  many  scheduled  through  1997  (Air  Force 
Medical  Information  Systems  Plan, 1989 ; 1 1 120-1 1 125 )  . 


Tri-Service  Pharmacy  System  (Tripharm) 

This  is  an  Air  Force,  Army,  and  Navy  based  initiative  to 
provide  pharmacies  with  complete  automation.  Tripharm 
provides  automated  data  processing  support  for  pharmacy 
operatins  in  the  following  areas:  census  registration, 
order  entry,  inventory  management,  doctor  database,  drug 
interaction  screening  and  allergy  notification,  UCA 
reporting,  drug  utilization  reports,  maximum  dose  screening, 
clinic  issues,  patient  profiles,  and  inventory  management 
and  ordering. 
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Bakercells  (an  automated  tablet  dispensing  system)  can 
be  interfaced  with  the  system.  The  AQCESS  system  has  been 
tested  and  can  act  as  an  interface  for  DEERS  checking. 
Modems  are  used  for  connecting  satellite  pharmacies. 

Tripharm  will  eventually  be  replaced  with  CHCS  (Air 
Force  Medical  Information  Systems  Plan,  1989 ; 1 1 1 39- 1 1 1 41 ) . 


Local  Area  Network  (LAW) 

A  local  area  network  is  a  datacommunications  system  that 
allows  many  independent  devices  (printers,  terminals,  other 
networks)  to  communicate  directly  within  a  moderately  sized 
geographic  area  over  a  communications  medium.  Networking 
allows  sharing  of  word  processing,  financial  analysis,  order 
processing,  and  many  other  applications  (Martin, 1989; 3-4) . 

Local  area  networks  are  required  for  many  of  the  systems 
mentioned  in  this  appendix. 


Medical  Expense  and  Performance  Reporting  System 
(EASIII/MEPRS) 

This  is  a  program  designed  to  report  unique  workload  and 
expense  data  for  each  free  standing  medical  facility  and 
dental  clinic  to  the  Office  of  the  Assistant  Secretary  of 
Defense  for  Health  Affairs.  It  has  interactive  capability 
with  MEDLOG.  Initial  installation  in  an  Air  Force  facility 
was  in  1983  and  there  are  no  formal  plans  for  future  of  this 
system  at  present  (Air  Force  Medical  Information  Systems 
Plan,  1989; III9-III10) . 


Wang  Patient  Appointment  and  Scheduling  (PAS) 

This  system  is  an  on-line  program  for  scheduling  patient 
appointments  in  the  Wang  operating  environment.  All  clinics 
are  able  to  schedule  appointments  and  provider  time  inde¬ 
pendently  and  print  daily  appointment  listings  for  manage¬ 
ment  purposes.  The  system  has  been  in  operation  at  several 
locations  throughout  the  Air  Force  since  approximately  1983. 
It  is  a  stand-alone  system  and  does  not  interface  with  any 
other  systems  (Newel  1 , 1992 ) . 


Automated  Quality  of  Care  Evaluation  Support  System  (AQCESS) 

The  quality  assurance  function  of  this  system  is  used  to 
monitor  the  quality  of  treatment  provided  at  medical  treat¬ 
ment  facilities.  Quality  assurance  enables  authorized  users 
to  identify  numerous  types  of  QA  indicators  and  provides  a 
standard,  documented  methodology  for  monitoring  these  items. 
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This  system  has  the  capability  to  interface  with  the 
Defense  Eligibility  Enrollment  System  (DEERS)  and  Tripharm. 
It  was  first  implemented  in  approximately  1985  and  is  in 
operation  at  virtually  all  Air  Force  medical  treatment 
facilities.  Its  functions  will  be  replaced  by  CHCS  (Air 
Force  Medical  Information  Systems  Plan,  1989;III62). 


Wang  Office 

This  is  an  electronic  mail  system  installed  on  the  local 
area  networks  at  several  medical  sites.  In  addition  to 
electronic  mail  and  messaging,  the  system  enables  users  to 
access  distributed  word  processing,  calendar,  and  other 
automated  office  applications  (Newel  1 , 1992 ) . 


Medical  Logistics  System  (MEDLOG) 

The  MEDLOG  System  is  an  integrated  peacetime  and  wartime 
logistics  system  that  includes  capabilities  to  control  and 
manage  medical  supplies,  medical  equipment,  quality  assur¬ 
ance  of  war  reserve  materiel ,  biomedical  equipment  mainte¬ 
nance  management,  excess  reporting  and  funds  control. 
Although  MEDLOG  was  initiated  as  two  separate  systems 
(MEDRAMS  and  MMMS-OL) ,  the  Air  Force  needed  to  operate  the 
same  way  in  peacetime  as  it  does  in  wartime.  MEDLOG  is  an 
on-line  interactive  system  on  a  distributive  mini-computer 
network  and  was  first  implemented  in  1989. 

The  system  has  operational  interfaces  with  the  Base 
Contracting  Automated  System  (BCAS),  Medical  Materiel 
Accounting  System  (MMAS ) ,  Medical  Expense  and  Reporting 
System  (MEPRS),  Base  Level  Autodin  Message  Extract  System 
(BLAMES),  Automated  Data  Report  Submission  System  (ADRESS), 
Military  Standard  Requisition  and  Issue  Procedure  System 
(MILSTRIP),  Medical.  Network  (MEDNET) ,  and  Defense  Data 
Network  ( DDN ) .  Planned  interfaces  include  the  Standard  Base 
Supply  System  (SBSS)  and  the  Tri-Service  Central  Processing 
and  Distribution  System  (CPD)  as  well  as  CHCS. 

The  MEDLOG  System  is  projected  to  be  replaced  by  a  new 
system  called  C-LOG  which  is  still  in  the  development  stages 
(Air  Force  Medical  Information  Systems  Plan,  1989;III59- 
1 1 1 61 ) . 


Personnel  Concept  III  (PCIII) 

The  PCIII  system  is  a  personnel  management  system 
designed  to  fully  automate  orderly  room  functions.  When 
fully  implemented  at  all  locations,  it  will  redistribute  and 
decentralize  current  Central  Base  Personnel  Office  functions 
out  to  the  orderly  rooms.  The  system  will  be  operational  in 


all  base  orderly  rooms,  including  those  in  the  medical 
treatment  facilities.  All  personnel  actions,  such  as 
military  record  entry,  locator  and  survivor  information,  and 
performance  evaluation  notification,  will  be  performed 
through  the  orderly  room  rather  than  the  central  office. 

The  system  is  still  being  implemented  or  planned  for 
implementation  at  many  Air  Force  bases.  It  will  be  con¬ 
nected  directly  on-line  with  mainframes  at  the  base  central 
offices  and  the  Military  Personnel  Center.  It  first  became 
operational  in  1988  (Air  Force  Medical  Information  Systems 
Plan,  1989; III114-III115)  . 


Tri-Service  Laboratory  System  (Trilab) 

The  Trilab  system  provides  automated  support  to  clinical 
laboratories.  It  supports  specimen  processing,  order  proc¬ 
essing,  results  reporting,  UCA  and  CAP  workload  tracking, 
quality  control  management,  and  other  administrative  func¬ 
tions.  The  system  has  the  capability  to  interface  with  a 
wide  variety  of  automated  laboratory  instrumentation. 

Modems  can  be  connected  to  the  central  processing  unit  for 
dial  up  diagnostics  and  software  downloads. 

The  twelfth  system  was  installed  in  late  1987  with  no 
further  proliferation  scheduled.  Its  contract  expired  as 
the  end  of  fiscal  year  1989  and  will  be  replaced  by  the 
laboratory  module  of  CHCS  (Air  Force  Medical  Information 
Systems  Plan,  1989 ;  1 1 1 36- 1 1 1 38 )  . 


Tri-Service  Radiology  System  (Trirad) 

This  system  performs  the  functions  of  automated  patient 
radiology  scheduling,  film  tracking  and  management,  proce¬ 
dure  result  and  management  reports,  thus  reducing  manual 
clerical  tasks  and  human  error,  produces  standardised  inter¬ 
face  for  radiology/pathology  reporting,  and  automates 
description,  diagnosis,  and  analysis  of  patient  radiographs. 

The  system  will  be  replaced  in  functionality  by  CHCS  and 
no  plans  have  been  made  for  its  proliferation  beyond  the 
four  Air  Forces  sites  where  it  is  presently  in  operation 
(Air  Force  Medical  Information  Systems  Plan,  1989;III42- 
III43)  . 
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